[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:54 dist-test-slave-hw7q krb5kdc[9584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:54 dist-test-slave-hw7q krb5kdc[9584](info): set up 2 sockets
Jun 19 10:25:54 dist-test-slave-hw7q krb5kdc[9584](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 10:25:55.162041 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:25:55.266582 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:44055
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:55.343605 9597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:55.343811 9597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:55.343839 9597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:55.344955 9597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:55.344995 9597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:55.345013 9597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:55.345028 9597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:55.345043 9597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:55.346485 9597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:44055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:55.346724 9597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:55.346920 9597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:25:55.351554 9597 server_base.cc:1034] running on GCE node
W20260619 10:25:55.351550 9605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:55.351585 9602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:55.351550 9603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:55.351949 9597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:55.352216 9597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:55.353355 9597 hybrid_clock.cc:648] HybridClock initialized: now 1781864755353342 us; error 30 us; skew 500 ppm
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:55.355647 9597 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:25:55.356057 9597 webserver.cc:460] Webserver started at http://127.9.90.126:40599/ using document root <none> and password file <none>
I20260619 10:25:55.356261 9597 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:55.356312 9597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:55.356426 9597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:55.357414 9597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "1b6df1fc215c44c9b04245b41b8b912a"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "3004a170ca2d3b848f7e015dff3b27a1"
server_key_iv: "84bf4bed930d63877b6675428268de67"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.357721 9597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "1b6df1fc215c44c9b04245b41b8b912a"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "3004a170ca2d3b848f7e015dff3b27a1"
server_key_iv: "84bf4bed930d63877b6675428268de67"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.359028 9597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:55.359702 9612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.359836 9597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:55.359917 9597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "1b6df1fc215c44c9b04245b41b8b912a"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "3004a170ca2d3b848f7e015dff3b27a1"
server_key_iv: "84bf4bed930d63877b6675428268de67"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.359977 9597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:55.387197 9597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:55.389191 9597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:55.389308 9597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:55.393013 9597 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:44055
I20260619 10:25:55.393065 9664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:44055 every 8 connection(s)
I20260619 10:25:55.393373 9597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:55.394268 9665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:55.396549 9665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a: Bootstrap starting.
I20260619 10:25:55.397188 9665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:55.397449 9665 log.cc:824] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:55.398110 9665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a: No bootstrap required, opened a new log
I20260619 10:25:55.399685 9665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6df1fc215c44c9b04245b41b8b912a" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44055 } }
I20260619 10:25:55.399824 9665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:55.399847 9665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6df1fc215c44c9b04245b41b8b912a, State: Initialized, Role: FOLLOWER
I20260619 10:25:55.399976 9665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6df1fc215c44c9b04245b41b8b912a" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44055 } }
I20260619 10:25:55.400036 9665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:55.400058 9665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:55.400104 9665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:55.400611 9665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6df1fc215c44c9b04245b41b8b912a" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44055 } }
I20260619 10:25:55.400707 9665 leader_election.cc:302] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b6df1fc215c44c9b04245b41b8b912a; no voters:
I20260619 10:25:55.400818 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 9597
I20260619 10:25:55.400859 9665 leader_election.cc:288] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:55.400889 9668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:55.400915 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:25:55.401007 9668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [term 1 LEADER]: Becoming Leader. State: Replica: 1b6df1fc215c44c9b04245b41b8b912a, State: Running, Role: LEADER
I20260619 10:25:55.401108 9665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:55.401087 9668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6df1fc215c44c9b04245b41b8b912a" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44055 } }
I20260619 10:25:55.401441 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b6df1fc215c44c9b04245b41b8b912a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6df1fc215c44c9b04245b41b8b912a" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44055 } } }
I20260619 10:25:55.401515 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:55.401564 9669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6df1fc215c44c9b04245b41b8b912a. Latest consensus state: current_term: 1 leader_uuid: "1b6df1fc215c44c9b04245b41b8b912a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6df1fc215c44c9b04245b41b8b912a" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44055 } } }
I20260619 10:25:55.401640 9669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:55.401683 9577 external_mini_cluster.cc:1428] Setting key 1a2e8b5ae00711aea5542b77d5110d8b
I20260619 10:25:55.401705 9674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:55.402256 9674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:55.403702 9674 catalog_manager.cc:1347] Generated new cluster ID: 914e141b50144dc8b0664a0db1d1336a
I20260619 10:25:55.403752 9674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:55.413951 9674 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:55.414652 9674 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:55.415797 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:55.402696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.9.90.126:44055)
0619 10:25:55.402995 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:55.403005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:55.403150 (+ 145us) server_negotiation.cc:407] Connection header received
0619 10:25:55.404917 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:55.404924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:55.404982 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:55.405099 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:55.406549 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.407461 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.408500 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.408909 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.409004 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:55.412272 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:55.412276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:55.412276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:55.412286 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:55.413890 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.414346 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.414348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.414349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.414433 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.414650 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.414651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.414652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.414914 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:55.415049 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:55.415310 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 10:25:55.415441 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":78,"threads_started":1}
I20260619 10:25:55.421304 9674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b6df1fc215c44c9b04245b41b8b912a: Generated new TSK 0
I20260619 10:25:55.421458 9674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:25:55.450131 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:44055
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:55.539047 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:55.539242 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:55.539275 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:55.540357 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:55.540397 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:55.540431 9691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:25:55.542045 9691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:44055
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:55.542311 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:55.542520 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:55.547276 9697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:55.547266 9699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:55.547262 9696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:55.547648 9691 server_base.cc:1034] running on GCE node
I20260619 10:25:55.547857 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:55.548089 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:55.549229 9691 hybrid_clock.cc:648] HybridClock initialized: now 1781864755549215 us; error 31 us; skew 500 ppm
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:55.551568 9691 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:25:55.551980 9691 webserver.cc:460] Webserver started at http://127.9.90.65:40663/ using document root <none> and password file <none>
I20260619 10:25:55.552147 9691 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:55.552183 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:55.552270 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:55.553213 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "333fb420852546208233abc5ec23d37e"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "a4def8d331d54b4976957238d32af8d6"
server_key_iv: "ed2960df72db0ad81161f8fbb5778314"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.553490 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "333fb420852546208233abc5ec23d37e"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "a4def8d331d54b4976957238d32af8d6"
server_key_iv: "ed2960df72db0ad81161f8fbb5778314"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.554698 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:55.555330 9706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.555510 9691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:55.555588 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "333fb420852546208233abc5ec23d37e"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "a4def8d331d54b4976957238d32af8d6"
server_key_iv: "ed2960df72db0ad81161f8fbb5778314"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.555653 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:55.565925 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:55.567883 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:55.568004 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:55.568189 9691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:55.568446 9691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:55.568477 9691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.568504 9691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:55.568528 9691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.574030 9691 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:37059
I20260619 10:25:55.574082 9819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:37059 every 8 connection(s)
I20260619 10:25:55.574407 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:25:55.574754 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 9691
I20260619 10:25:55.574842 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:25:55.574987 9577 external_mini_cluster.cc:1428] Setting key 8ef4d2f91bff61635cbf5812f900d2fc
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
I20260619 10:25:55.583750 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:55.575070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:33971 (local address 127.9.90.126:44055)
0619 10:25:55.575124 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:55.575126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:55.575372 (+ 246us) server_negotiation.cc:407] Connection header received
0619 10:25:55.575935 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:55.575937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:55.575948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:55.576010 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:55.576933 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.577419 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.578777 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.578979 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.579030 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:55.581276 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:55.581277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:55.581277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:55.581284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:55.582570 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.582930 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.582930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.582930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.582973 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.583183 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.583185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.583185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.583303 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:55.583382 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:55.583612 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 10:25:55.583684 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:25:55.583878 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:55.575043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44055 (local address 127.9.90.65:33971)
0619 10:25:55.575264 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:55.575307 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:55.575846 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:55.576062 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:55.576066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:55.576576 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:55.576849 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:55.576855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.577497 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.577498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:55.578692 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:55.578695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.579053 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.579054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:55.579192 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:55.579654 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:55.579663 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:55.581192 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:55.582641 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:55.582647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:55.582650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:55.582869 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:55.583015 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:55.583016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:55.583016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:55.583099 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:55.583383 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:55.583389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:55.583504 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 10:25:55.583586 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 10:25:55.583710 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260619 10:25:55.584343 9820 heartbeater.cc:347] Connected to a master server at 127.9.90.126:44055
I20260619 10:25:55.584435 9820 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:55.584596 9820 heartbeater.cc:511] Master 127.9.90.126:44055 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
I20260619 10:25:55.585147 9629 ts_manager.cc:194] Registered new tserver with Master: 333fb420852546208233abc5ec23d37e (127.9.90.65:37059)
I20260619 10:25:55.585768 9629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:33971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:25:55.600344 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:44055
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:55.679451 9825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:55.679653 9825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:55.679685 9825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:55.680812 9825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:55.680852 9825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:55.680889 9825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:25:55.682312 9825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:44055
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:55.682538 9825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:55.682739 9825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:55.687286 9830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:55.687373 9831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:55.687464 9833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:55.687702 9825 server_base.cc:1034] running on GCE node
I20260619 10:25:55.687891 9825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:55.688105 9825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:55.689255 9825 hybrid_clock.cc:648] HybridClock initialized: now 1781864755689230 us; error 35 us; skew 500 ppm
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:55.691604 9825 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:25:55.692039 9825 webserver.cc:460] Webserver started at http://127.9.90.66:38357/ using document root <none> and password file <none>
I20260619 10:25:55.692238 9825 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:55.692286 9825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:55.692416 9825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:55.693414 9825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "216caeccbfa240b783008b04a2342321"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "eba234aa2c7b98229cb2050e0d939587"
server_key_iv: "ed021c6322974131a10eb62656f54faa"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.693740 9825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "216caeccbfa240b783008b04a2342321"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "eba234aa2c7b98229cb2050e0d939587"
server_key_iv: "ed021c6322974131a10eb62656f54faa"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.695046 9825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:55.695834 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.696009 9825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:55.696096 9825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "216caeccbfa240b783008b04a2342321"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "eba234aa2c7b98229cb2050e0d939587"
server_key_iv: "ed021c6322974131a10eb62656f54faa"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.696197 9825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:55.723491 9825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:55.726001 9825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:55.726203 9825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:55.726471 9825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:55.726831 9825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:55.726876 9825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.726905 9825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:55.726935 9825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.732527 9825 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:40343
I20260619 10:25:55.732602 9953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:40343 every 8 connection(s)
I20260619 10:25:55.732862 9825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:25:55.735286 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 9825
I20260619 10:25:55.735390 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:25:55.735553 9577 external_mini_cluster.cc:1428] Setting key c1881e800651b208b6982f2427b9bfad
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:55.742383 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:55.733546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:39825 (local address 127.9.90.126:44055)
0619 10:25:55.733592 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:55.733594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:55.733768 (+ 174us) server_negotiation.cc:407] Connection header received
0619 10:25:55.734280 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:55.734281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:55.734291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:55.734343 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:55.735346 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.735998 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.737077 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.737335 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.737397 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:55.739595 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:55.739596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:55.739597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:55.739604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:55.740983 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.741387 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.741389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.741389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.741447 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.741734 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.741735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.741735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.741893 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:55.741963 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:55.742207 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 10:25:55.742290 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:55.742451 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:55.733468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44055 (local address 127.9.90.66:39825)
0619 10:25:55.733656 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:55.733692 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:55.734192 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:55.734402 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:55.734406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:55.734943 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:55.735129 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:55.735134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.736078 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.736079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:55.737003 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:55.737006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.737430 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.737431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:55.737554 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:55.738025 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:55.738033 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:55.739509 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:55.741076 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:55.741081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:55.741083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:55.741318 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:55.741538 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:55.741539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:55.741539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:55.741662 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:55.741959 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:55.741965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:55.742081 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 10:25:55.742169 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 10:25:55.742282 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
I20260619 10:25:55.742871 9954 heartbeater.cc:347] Connected to a master server at 127.9.90.126:44055
I20260619 10:25:55.742969 9954 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
I20260619 10:25:55.743155 9954 heartbeater.cc:511] Master 127.9.90.126:44055 requested a full tablet report, sending...
I20260619 10:25:55.743503 9629 ts_manager.cc:194] Registered new tserver with Master: 216caeccbfa240b783008b04a2342321 (127.9.90.66:40343)
I20260619 10:25:55.743885 9629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:39825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:25:55.757972 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:44055
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:55.835210 9959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:55.835397 9959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:55.835427 9959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:55.836505 9959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:55.836544 9959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:55.836577 9959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:25:55.838007 9959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:44055
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:55.838217 9959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:55.838418 9959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:25:55.842957 9959 server_base.cc:1034] running on GCE node
W20260619 10:25:55.842921 9967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:55.843044 9964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:55.843122 9965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:55.843353 9959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:55.843585 9959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:55.844730 9959 hybrid_clock.cc:648] HybridClock initialized: now 1781864755844715 us; error 32 us; skew 500 ppm
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:55.847081 9959 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:25:55.847472 9959 webserver.cc:460] Webserver started at http://127.9.90.67:35109/ using document root <none> and password file <none>
I20260619 10:25:55.847673 9959 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:55.847718 9959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:55.847843 9959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:55.848810 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "59de9bcdf8914cb2846f57761b9da56c"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "4ddf5c7bcee65c0abcde777c9b7cc115"
server_key_iv: "3523a80db16c8c112ec1b6497121a3ad"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.849131 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "59de9bcdf8914cb2846f57761b9da56c"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "4ddf5c7bcee65c0abcde777c9b7cc115"
server_key_iv: "3523a80db16c8c112ec1b6497121a3ad"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.850338 9959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 10:25:55.850992 9974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.851130 9959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.851207 9959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "59de9bcdf8914cb2846f57761b9da56c"
format_stamp: "Formatted at 2026-06-19 10:25:55 on dist-test-slave-hw7q"
server_key: "4ddf5c7bcee65c0abcde777c9b7cc115"
server_key_iv: "3523a80db16c8c112ec1b6497121a3ad"
server_key_version: "encryptionkey@0"
I20260619 10:25:55.851261 9959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:55.860647 9959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:55.862649 9959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:55.862764 9959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:55.862952 9959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:55.863211 9959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:55.863250 9959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.863278 9959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:55.863302 9959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:55.869060 9959 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:40899
I20260619 10:25:55.869094 10087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:40899 every 8 connection(s)
I20260619 10:25:55.869371 9959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:25:55.872051 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 9959
I20260619 10:25:55.872128 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:25:55.872279 9577 external_mini_cluster.cc:1428] Setting key 67f57651e4cc762096f45d56b156eb3f
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:55.878328 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:55.870015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:42351 (local address 127.9.90.126:44055)
0619 10:25:55.870064 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:55.870065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:55.870264 (+ 199us) server_negotiation.cc:407] Connection header received
0619 10:25:55.870754 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:55.870756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:55.870767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:55.870814 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:55.871721 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.872267 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.873542 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.873747 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.873852 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:55.875710 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:55.875711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:55.875712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:55.875718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:55.876878 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.877288 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.877289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.877289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.877350 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.877581 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.877582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.877582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.877701 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:55.877833 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:55.878138 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 10:25:55.878225 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:25:55.878439 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:55.869999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44055 (local address 127.9.90.67:42351)
0619 10:25:55.870157 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:55.870194 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:55.870665 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:55.870894 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:55.870898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:55.871436 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:55.871632 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:55.871636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.872362 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.872363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:55.873469 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:55.873473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.873838 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.873839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:55.873965 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:55.874398 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:55.874406 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:55.875630 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:55.876977 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:55.876984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:55.876986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:55.877214 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:55.877408 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:55.877409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:55.877410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:55.877519 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:55.877827 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:55.877833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:55.877971 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 10:25:55.878066 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 10:25:55.878225 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260619 10:25:55.878858 10088 heartbeater.cc:347] Connected to a master server at 127.9.90.126:44055
I20260619 10:25:55.878940 10088 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:55.879096 10088 heartbeater.cc:511] Master 127.9.90.126:44055 requested a full tablet report, sending...
I20260619 10:25:55.879402 9629 ts_manager.cc:194] Registered new tserver with Master: 59de9bcdf8914cb2846f57761b9da56c (127.9.90.67:40899)
I20260619 10:25:55.879766 9629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:42351
I20260619 10:25:55.884209 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:25:55.905002 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:55.898101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.9.90.65:37059)
0619 10:25:55.898282 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:55.898291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:55.898301 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:25:55.898345 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:55.898347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:55.898361 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:55.898424 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:55.898940 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.899644 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.900356 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.900724 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.900776 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:55.902839 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:55.902845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:55.902846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:55.902852 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:55.904162 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.904438 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.904441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.904441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.904490 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.904606 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.904606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.904606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.904712 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:55.904817 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:55.904851 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 10:25:55.904910 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
W20260619 10:25:55.905376 9734 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32852
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[9584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:25:55.925032 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:55.919062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.9.90.65:37059)
0619 10:25:55.919117 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:55.919118 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:55.919148 (+ 30us) server_negotiation.cc:407] Connection header received
0619 10:25:55.919209 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:55.919210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:55.919221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:55.919265 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:55.919695 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.920157 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.920796 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:55.920997 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:55.921043 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:55.923057 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:55.923059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:55.923059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:55.923065 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:55.924105 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.924386 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.924387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.924388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.924437 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:55.924593 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:55.924594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:55.924594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:55.924679 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:55.924777 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:55.924871 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 10:25:55.924945 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:25:55.925556 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 9691
I20260619 10:25:55.930357 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 9825
I20260619 10:25:55.933868 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 9959
I20260619 10:25:55.937279 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 9597
2026-06-19T10:25:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1743 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[10102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[10102](info): set up 2 sockets
Jun 19 10:25:55 dist-test-slave-hw7q krb5kdc[10102](info): commencing operation
krb5kdc: starting...
W20260619 10:25:56.884990 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:25:56 dist-test-slave-hw7q krb5kdc[10102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:25:56.987960 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:39447
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:57.066403 10115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:57.066596 10115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:57.066617 10115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:57.067682 10115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:57.067714 10115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:57.067727 10115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:57.067739 10115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:57.067749 10115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:57.069099 10115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:57.069283 10115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:57.069448 10115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:57.073829 10123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:57.073853 10121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:57.073932 10115 server_base.cc:1034] running on GCE node
W20260619 10:25:57.073895 10120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:57.074222 10115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:57.074501 10115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:57.075645 10115 hybrid_clock.cc:648] HybridClock initialized: now 1781864757075629 us; error 31 us; skew 500 ppm
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:57.077909 10115 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:25:57.078297 10115 webserver.cc:460] Webserver started at http://127.9.90.126:45545/ using document root <none> and password file <none>
I20260619 10:25:57.078511 10115 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:57.078553 10115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:57.078676 10115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:57.079684 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "ac01408f38d9955f6250819d69d66a19"
server_key_iv: "919e289f02554e661be1ec305feb0a64"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.079996 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "ac01408f38d9955f6250819d69d66a19"
server_key_iv: "919e289f02554e661be1ec305feb0a64"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.081156 10115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:25:57.081822 10130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.081972 10115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:57.082054 10115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "ac01408f38d9955f6250819d69d66a19"
server_key_iv: "919e289f02554e661be1ec305feb0a64"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.082105 10115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:57.094767 10115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:57.096738 10115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:57.096844 10115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:57.100277 10115 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:39447
I20260619 10:25:57.100306 10182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:39447 every 8 connection(s)
I20260619 10:25:57.100651 10115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:57.101403 10183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:57.101824 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 10115
I20260619 10:25:57.101913 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:25:57.102067 9577 external_mini_cluster.cc:1428] Setting key 862b6aa512f3bf75487aabb743fc4033
I20260619 10:25:57.104166 10183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19: Bootstrap starting.
I20260619 10:25:57.104836 10183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:57.105098 10183 log.cc:824] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:57.105743 10183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19: No bootstrap required, opened a new log
I20260619 10:25:57.107774 10183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39447 } }
I20260619 10:25:57.107913 10183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:57.107946 10183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf4e9fab17a4470a71e5b96b0ee4f19, State: Initialized, Role: FOLLOWER
I20260619 10:25:57.108067 10183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39447 } }
I20260619 10:25:57.108139 10183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:57.108168 10183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:57.108214 10183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:57.108754 10183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39447 } }
I20260619 10:25:57.108865 10183 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cf4e9fab17a4470a71e5b96b0ee4f19; no voters:
I20260619 10:25:57.109026 10183 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:57.109069 10188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:57.109282 10183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:57.109469 10188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [term 1 LEADER]: Becoming Leader. State: Replica: 2cf4e9fab17a4470a71e5b96b0ee4f19, State: Running, Role: LEADER
I20260619 10:25:57.109556 10188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39447 } }
I20260619 10:25:57.110301 10189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39447 } } }
I20260619 10:25:57.110378 10189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:57.110561 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf4e9fab17a4470a71e5b96b0ee4f19. Latest consensus state: current_term: 1 leader_uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf4e9fab17a4470a71e5b96b0ee4f19" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39447 } } }
I20260619 10:25:57.110785 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19 [sys.catalog]: This master's current role is: LEADER
W20260619 10:25:57.111021 10203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:25:57.111078 10203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:25:57.111246 10202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:25:57.111392 10202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:57.112221 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.102526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.9.90.126:39447)
0619 10:25:57.102692 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.102702 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.102736 (+ 34us) server_negotiation.cc:407] Connection header received
0619 10:25:57.103220 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.103229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.103286 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.103447 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:57.103939 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.105207 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.106055 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.106658 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.106763 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.108819 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:57.108824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:57.108824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:57.108836 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:57.110763 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.111147 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.111150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.111150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.111237 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.111416 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.111417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.111417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.111642 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:57.111777 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.111906 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.112031 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260619 10:25:57.112843 10202 catalog_manager.cc:1347] Generated new cluster ID: 13bf0bd1dad54818acfab3a2aaa2665b
I20260619 10:25:57.112890 10202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:57.122467 10202 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:57.122921 10202 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:57.126050 10202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cf4e9fab17a4470a71e5b96b0ee4f19: Generated new TSK 0
I20260619 10:25:57.126176 10202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:25:57.150233 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39447
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:57.227459 10209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:57.227640 10209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:57.227662 10209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:57.228809 10209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:57.228849 10209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:57.228874 10209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:25:57.230309 10209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:57.230515 10209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:57.230705 10209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:57.235157 10215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:57.235164 10214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:57.235176 10217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:57.235533 10209 server_base.cc:1034] running on GCE node
I20260619 10:25:57.235738 10209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:57.235932 10209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:57.237057 10209 hybrid_clock.cc:648] HybridClock initialized: now 1781864757237047 us; error 29 us; skew 500 ppm
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:57.239305 10209 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:25:57.239707 10209 webserver.cc:460] Webserver started at http://127.9.90.65:33173/ using document root <none> and password file <none>
I20260619 10:25:57.239918 10209 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:57.239962 10209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:57.240049 10209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:57.240993 10209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "e2e1b8fd28e1366303fad089fb98c888"
server_key_iv: "08a5f274596ec646b1032a9c2802f588"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.241269 10209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "e2e1b8fd28e1366303fad089fb98c888"
server_key_iv: "08a5f274596ec646b1032a9c2802f588"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.242451 10209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:57.243053 10224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.243196 10209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:57.243265 10209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "e2e1b8fd28e1366303fad089fb98c888"
server_key_iv: "08a5f274596ec646b1032a9c2802f588"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.243328 10209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:57.258349 10209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:57.259938 10209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:57.260061 10209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:57.260259 10209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:57.260526 10209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:57.260557 10209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.260579 10209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:57.260602 10209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.265949 10209 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:35601
I20260619 10:25:57.265996 10337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:35601 every 8 connection(s)
I20260619 10:25:57.266276 10209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:57.274279 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 10209
I20260619 10:25:57.274360 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:25:57.274493 9577 external_mini_cluster.cc:1428] Setting key c8cb92d702cb1c4929d0faa3d1b2e2a2
I20260619 10:25:57.274875 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.266897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:60413 (local address 127.9.90.126:39447)
0619 10:25:57.266955 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.266957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.267127 (+ 170us) server_negotiation.cc:407] Connection header received
0619 10:25:57.268031 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.268032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.268044 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.268094 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:57.268704 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.269204 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.270243 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.270447 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.270509 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.272570 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:57.272571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:57.272572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:57.272578 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:57.273609 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.273975 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.273976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.273976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.274022 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.274253 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.274254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.274254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.274350 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:57.274407 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.274703 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.274809 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:25:57.275031 10340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.266876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:39447 (local address 127.9.90.65:60413)
0619 10:25:57.267013 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:57.267048 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:57.267923 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:57.268163 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:57.268167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:57.268428 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:57.268605 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.268610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.269286 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.269287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.270114 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.270117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.270654 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.270655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.270754 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.271185 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:57.271192 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:57.272490 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:57.273670 (+ 1180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:57.273675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:57.273676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:57.273923 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:57.274069 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:57.274069 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:57.274070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:57.274194 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:57.274419 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:57.274425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:57.274560 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 10:25:57.274670 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 10:25:57.274823 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260619 10:25:57.275450 10338 heartbeater.cc:347] Connected to a master server at 127.9.90.126:39447
I20260619 10:25:57.275537 10338 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:57.275699 10338 heartbeater.cc:511] Master 127.9.90.126:39447 requested a full tablet report, sending...
I20260619 10:25:57.276165 10147 ts_manager.cc:194] Registered new tserver with Master: bf4807ccdd454e42a16e37cc8b6b0fcb (127.9.90.65:35601)
I20260619 10:25:57.276738 10147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:60413
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:25:57.294354 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39447
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:57.371505 10343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:57.371701 10343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:57.371724 10343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:57.372803 10343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:57.372835 10343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:57.372859 10343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:25:57.374290 10343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:57.374469 10343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:57.374631 10343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:57.379208 10351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:57.379190 10349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:57.379189 10348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:57.379434 10343 server_base.cc:1034] running on GCE node
I20260619 10:25:57.379634 10343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:57.379868 10343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:57.381007 10343 hybrid_clock.cc:648] HybridClock initialized: now 1781864757380989 us; error 28 us; skew 500 ppm
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:57.383582 10343 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:25:57.383998 10343 webserver.cc:460] Webserver started at http://127.9.90.66:33917/ using document root <none> and password file <none>
I20260619 10:25:57.384219 10343 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:57.384267 10343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:57.384393 10343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:57.385462 10343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "4235da6d5b79426da92def935adf139a"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "43c60282bac4e742d78f22617beca73b"
server_key_iv: "46c69a9a380930ab43b904c2b874f467"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.385839 10343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "4235da6d5b79426da92def935adf139a"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "43c60282bac4e742d78f22617beca73b"
server_key_iv: "46c69a9a380930ab43b904c2b874f467"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.387077 10343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:57.387775 10358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.387928 10343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:57.388011 10343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "4235da6d5b79426da92def935adf139a"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "43c60282bac4e742d78f22617beca73b"
server_key_iv: "46c69a9a380930ab43b904c2b874f467"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.388077 10343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:57.408015 10343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:57.410220 10343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:57.410358 10343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:57.410559 10343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:57.410909 10343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:57.410950 10343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.410979 10343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:57.411001 10343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.416177 10343 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:41635
I20260619 10:25:57.416229 10471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:41635 every 8 connection(s)
I20260619 10:25:57.416503 10343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:25:57.418241 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 10343
I20260619 10:25:57.418330 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:25:57.418493 9577 external_mini_cluster.cc:1428] Setting key 69ec28a890eecd68fda5084b51c68d11
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:57.425772 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.417120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:44985 (local address 127.9.90.126:39447)
0619 10:25:57.417175 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.417177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.417400 (+ 223us) server_negotiation.cc:407] Connection header received
0619 10:25:57.418306 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.418308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.418320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.418390 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:57.419164 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.419646 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.420835 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.421025 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.421078 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.423305 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:57.423306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:57.423306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:57.423318 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:57.424496 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.424935 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.424935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.424936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.424978 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.425209 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.425210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.425210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.425298 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:57.425355 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.425653 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.425709 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:25:57.426044 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.417088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:39447 (local address 127.9.90.66:44985)
0619 10:25:57.417254 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:57.417294 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:57.418215 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:57.418527 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:57.418532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:57.418863 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:57.419089 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.419096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.419728 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.419729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.420759 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.420762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.421090 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.421091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.421218 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.421629 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:57.421638 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:57.423218 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:57.424569 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:57.424576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:57.424579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:57.424869 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:57.425031 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:57.425032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:57.425032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:57.425153 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:57.425365 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:57.425370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:57.425527 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 10:25:57.425638 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 10:25:57.425830 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
I20260619 10:25:57.426491 10472 heartbeater.cc:347] Connected to a master server at 127.9.90.126:39447
I20260619 10:25:57.426573 10472 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:57.426733 10472 heartbeater.cc:511] Master 127.9.90.126:39447 requested a full tablet report, sending...
I20260619 10:25:57.427098 10147 ts_manager.cc:194] Registered new tserver with Master: 4235da6d5b79426da92def935adf139a (127.9.90.66:41635)
I20260619 10:25:57.427438 10147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:44985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:25:57.440213 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39447
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:57.516916 10477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:57.517095 10477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:57.517118 10477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:57.518234 10477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:57.518268 10477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:57.518293 10477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:25:57.519685 10477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:57.519853 10477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:57.520025 10477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:57.524562 10485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:57.524567 10483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:57.524566 10482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:57.524767 10477 server_base.cc:1034] running on GCE node
I20260619 10:25:57.525012 10477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:57.525205 10477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:57.526333 10477 hybrid_clock.cc:648] HybridClock initialized: now 1781864757526319 us; error 31 us; skew 500 ppm
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:57.528556 10477 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:25:57.528937 10477 webserver.cc:460] Webserver started at http://127.9.90.67:39853/ using document root <none> and password file <none>
I20260619 10:25:57.529116 10477 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:57.529155 10477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:57.529238 10477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:57.530205 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "9067360f2cb64f26b16b8b2ae95814ce"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "8b51d2b32ed7fbfa603410bae667cd99"
server_key_iv: "ae8ce75bba272c1cc2cb8c44f62dfd2a"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.530481 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "9067360f2cb64f26b16b8b2ae95814ce"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "8b51d2b32ed7fbfa603410bae667cd99"
server_key_iv: "ae8ce75bba272c1cc2cb8c44f62dfd2a"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.531567 10477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:25:57.532166 10492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.532316 10477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:57.532388 10477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "9067360f2cb64f26b16b8b2ae95814ce"
format_stamp: "Formatted at 2026-06-19 10:25:57 on dist-test-slave-hw7q"
server_key: "8b51d2b32ed7fbfa603410bae667cd99"
server_key_iv: "ae8ce75bba272c1cc2cb8c44f62dfd2a"
server_key_version: "encryptionkey@0"
I20260619 10:25:57.532439 10477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:57.544617 10477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:57.546440 10477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:57.546541 10477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:57.546707 10477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:25:57.546947 10477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:57.546975 10477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.546994 10477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:57.547011 10477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:57.552125 10477 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:38153
I20260619 10:25:57.552184 10605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:38153 every 8 connection(s)
I20260619 10:25:57.552451 10477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:25:57.554227 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 10477
I20260619 10:25:57.554322 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:25:57.554508 9577 external_mini_cluster.cc:1428] Setting key a17bf89904fdd1d04a1e3a90cc4de7b3
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:57.560843 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.553159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:57111 (local address 127.9.90.126:39447)
0619 10:25:57.553213 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.553215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.553390 (+ 175us) server_negotiation.cc:407] Connection header received
0619 10:25:57.554388 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.554390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.554400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.554466 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:57.555160 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.555682 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.556611 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.556794 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.556838 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.558791 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:57.558791 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:57.558792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:57.558798 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:57.559862 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.560180 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.560180 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.560180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.560224 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.560405 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.560405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.560406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.560494 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:57.560536 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.560725 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.560779 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:25:57.561023 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.553130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:39447 (local address 127.9.90.67:57111)
0619 10:25:57.553282 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:57.553315 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:57.554286 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:57.554537 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:57.554542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:57.554828 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:57.555079 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.555085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.555754 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.555754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.556543 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.556545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.556870 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.556871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.556975 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.557413 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:57.557421 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:57.558708 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:57.559924 (+ 1216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:57.559929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:57.559931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:57.560127 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:57.560278 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:57.560279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:57.560279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:57.560365 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:57.560533 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:57.560535 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:57.560636 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 10:25:57.560708 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 10:25:57.560827 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260619 10:25:57.561398 10606 heartbeater.cc:347] Connected to a master server at 127.9.90.126:39447
I20260619 10:25:57.561477 10606 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:57.561635 10606 heartbeater.cc:511] Master 127.9.90.126:39447 requested a full tablet report, sending...
I20260619 10:25:57.562011 10147 ts_manager.cc:194] Registered new tserver with Master: 9067360f2cb64f26b16b8b2ae95814ce (127.9.90.67:38153)
I20260619 10:25:57.562345 10147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:57111
I20260619 10:25:57.566061 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:25:57.572664 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.567500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.9.90.126:39447)
0619 10:25:57.567548 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.567550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.567641 (+ 91us) server_negotiation.cc:407] Connection header received
0619 10:25:57.567700 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.567701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.567711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.567755 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:57.568267 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.568725 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.569404 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.569582 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.569633 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.570662 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:57.570663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:57.570663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:57.570669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:57.571675 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.571967 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.571967 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.571968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.572016 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.572170 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.572171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.572172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.572303 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:57.572394 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.572514 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.572578 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:25:57.574560 10147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:25:57.576004 10147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:57.581830 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.577868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:35601 (local address 127.0.0.1:41826)
0619 10:25:57.578110 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:57.578119 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:57.578167 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:57.578676 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:57.578677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:57.578679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:57.578797 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.578800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.579956 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.579958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.580710 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.580713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.581507 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.581508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.581582 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.581596 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:25:57.581669 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 10:25:57.581720 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":20,"thread_start_us":85,"threads_started":1}
I20260619 10:25:57.582039 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.577862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38153 (local address 127.0.0.1:36782)
0619 10:25:57.577981 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:57.578019 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:57.578077 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:57.578447 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:57.578450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:57.578452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:57.578516 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.578518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.579754 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.579755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.581019 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.581022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.581688 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.581689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.581876 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.581880 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:57.581929 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:25:57.581970 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":41,"thread_start_us":73,"threads_started":1}
I20260619 10:25:57.582134 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.577896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:41635 (local address 127.0.0.1:55098)
0619 10:25:57.578479 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:57.578488 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:57.578538 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:57.578675 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:57.578676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:57.578678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:57.578743 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.578745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.579702 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.579703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.581019 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.581022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.581894 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.581895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.581953 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.581958 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:25:57.581997 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 10:25:57.582037 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":36,"thread_start_us":66,"threads_started":1}
I20260619 10:25:57.582301 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.578113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.9.90.66:41635)
0619 10:25:57.578305 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.578308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.578541 (+ 233us) server_negotiation.cc:407] Connection header received
0619 10:25:57.578590 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.578593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.578606 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.578673 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:57.578854 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.579613 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.581102 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.581804 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.582081 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.582150 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.582179 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.582216 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260619 10:25:57.582310 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.578111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.9.90.67:38153)
0619 10:25:57.578279 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.578284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.578293 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:57.578327 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.578329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.578345 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.578424 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:57.578606 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.579674 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.581115 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.581609 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.581677 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.581728 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.582201 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.582240 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260619 10:25:57.582502 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.578296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.9.90.65:35601)
0619 10:25:57.578463 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.578467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.578476 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:57.578509 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.578511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.578525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.578592 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:57.578878 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.579879 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.580800 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.581428 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.581495 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.581549 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.582406 (+ 857us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.582447 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260619 10:25:57.583062 10406 tablet_service.cc:1467] Processing CreateTablet for tablet 772c024789054896b6916a6d4daf99f6 (DEFAULT_TABLE table=test-table [id=2a7cefe7ac1f4c5da2c6faf7a24f5179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:57.583062 10540 tablet_service.cc:1467] Processing CreateTablet for tablet 772c024789054896b6916a6d4daf99f6 (DEFAULT_TABLE table=test-table [id=2a7cefe7ac1f4c5da2c6faf7a24f5179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:57.583389 10406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772c024789054896b6916a6d4daf99f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:57.583389 10540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772c024789054896b6916a6d4daf99f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:57.585861 10620 tablet_bootstrap.cc:492] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a: Bootstrap starting.
I20260619 10:25:57.585942 10621 tablet_bootstrap.cc:492] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce: Bootstrap starting.
I20260619 10:25:57.586341 10272 tablet_service.cc:1467] Processing CreateTablet for tablet 772c024789054896b6916a6d4daf99f6 (DEFAULT_TABLE table=test-table [id=2a7cefe7ac1f4c5da2c6faf7a24f5179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:57.586473 10620 tablet_bootstrap.cc:654] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:57.586547 10621 tablet_bootstrap.cc:654] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:57.586653 10272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772c024789054896b6916a6d4daf99f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:57.586983 10621 log.cc:824] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:57.586984 10620 log.cc:824] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:57.587569 10621 tablet_bootstrap.cc:492] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce: No bootstrap required, opened a new log
I20260619 10:25:57.587625 10621 ts_tablet_manager.cc:1397] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:57.588989 10624 tablet_bootstrap.cc:492] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb: Bootstrap starting.
I20260619 10:25:57.588923 10621 raft_consensus.cc:348] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } }
I20260619 10:25:57.589088 10621 raft_consensus.cc:374] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:57.589124 10621 raft_consensus.cc:729] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9067360f2cb64f26b16b8b2ae95814ce, State: Initialized, Role: FOLLOWER
I20260619 10:25:57.589344 10621 consensus_queue.cc:260] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } }
I20260619 10:25:57.589625 10621 ts_tablet_manager.cc:1428] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:57.589713 10620 tablet_bootstrap.cc:492] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a: No bootstrap required, opened a new log
I20260619 10:25:57.589740 10606 heartbeater.cc:503] Master 127.9.90.126:39447 was elected leader, sending a full tablet report...
I20260619 10:25:57.589767 10624 tablet_bootstrap.cc:654] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:57.589778 10620 ts_tablet_manager.cc:1397] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 10:25:57.590310 10624 log.cc:824] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:57.591039 10624 tablet_bootstrap.cc:492] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb: No bootstrap required, opened a new log
I20260619 10:25:57.591086 10624 ts_tablet_manager.cc:1397] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:57.591599 10620 raft_consensus.cc:348] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } }
I20260619 10:25:57.591727 10620 raft_consensus.cc:374] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:57.591758 10620 raft_consensus.cc:729] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4235da6d5b79426da92def935adf139a, State: Initialized, Role: FOLLOWER
I20260619 10:25:57.591857 10620 consensus_queue.cc:260] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } }
I20260619 10:25:57.592092 10620 ts_tablet_manager.cc:1428] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:25:57.592120 10472 heartbeater.cc:503] Master 127.9.90.126:39447 was elected leader, sending a full tablet report...
I20260619 10:25:57.592604 10624 raft_consensus.cc:348] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } }
I20260619 10:25:57.592772 10624 raft_consensus.cc:374] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:57.592808 10624 raft_consensus.cc:729] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4807ccdd454e42a16e37cc8b6b0fcb, State: Initialized, Role: FOLLOWER
I20260619 10:25:57.592931 10624 consensus_queue.cc:260] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } }
I20260619 10:25:57.593159 10624 ts_tablet_manager.cc:1428] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:57.593181 10338 heartbeater.cc:503] Master 127.9.90.126:39447 was elected leader, sending a full tablet report...
I20260619 10:25:57.603084 10627 raft_consensus.cc:482] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:57.603179 10627 raft_consensus.cc:504] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } }
I20260619 10:25:57.603474 10627 leader_election.cc:288] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9067360f2cb64f26b16b8b2ae95814ce (127.9.90.67:38153), bf4807ccdd454e42a16e37cc8b6b0fcb (127.9.90.65:35601)
I20260619 10:25:57.606566 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.603693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:40847 (local address 127.9.90.65:35601)
0619 10:25:57.603738 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.603740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.603748 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:25:57.603770 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.603771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.603782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.603833 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:57.603993 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.604723 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.605772 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.606254 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.606309 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.606326 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.606467 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.606502 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:25:57.606562 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.603606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:35601 (local address 127.9.90.66:40847)
0619 10:25:57.603649 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:57.603659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:57.603694 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:57.603850 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:57.603851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:57.603854 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:57.603922 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.603924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.604791 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.604792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.605693 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.605703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.606329 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.606330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.606404 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.606407 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:57.606458 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:25:57.606494 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":25}
I20260619 10:25:57.606639 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.603715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:41953 (local address 127.9.90.67:38153)
0619 10:25:57.603754 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.603756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.603974 (+ 218us) server_negotiation.cc:407] Connection header received
0619 10:25:57.604027 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.604028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.604037 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.604084 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:57.604177 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.604872 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.605956 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.606331 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.606399 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.606408 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.606543 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.606583 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:57.606666 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.603606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38153 (local address 127.9.90.66:41953)
0619 10:25:57.603908 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:57.603918 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:57.603980 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:57.604089 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:57.604089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:57.604090 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:57.604139 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.604140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.604948 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.604949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.605870 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:57.605874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.606409 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.606410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:57.606477 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.606480 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:57.606531 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:25:57.606568 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":131,"thread_start_us":108,"threads_started":1}
I20260619 10:25:57.606943 10560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772c024789054896b6916a6d4daf99f6" candidate_uuid: "4235da6d5b79426da92def935adf139a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9067360f2cb64f26b16b8b2ae95814ce" is_pre_election: true
I20260619 10:25:57.606994 10292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772c024789054896b6916a6d4daf99f6" candidate_uuid: "4235da6d5b79426da92def935adf139a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" is_pre_election: true
I20260619 10:25:57.607085 10560 raft_consensus.cc:2454] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4235da6d5b79426da92def935adf139a in term 0.
I20260619 10:25:57.607111 10292 raft_consensus.cc:2454] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4235da6d5b79426da92def935adf139a in term 0.
I20260619 10:25:57.607282 10360 leader_election.cc:302] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4235da6d5b79426da92def935adf139a, 9067360f2cb64f26b16b8b2ae95814ce; no voters:
I20260619 10:25:57.607403 10627 raft_consensus.cc:2781] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:57.607465 10627 raft_consensus.cc:482] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:57.607481 10627 raft_consensus.cc:3037] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:57.608096 10627 raft_consensus.cc:504] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } }
I20260619 10:25:57.608209 10627 leader_election.cc:288] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [CANDIDATE]: Term 1 election: Requested vote from peers 9067360f2cb64f26b16b8b2ae95814ce (127.9.90.67:38153), bf4807ccdd454e42a16e37cc8b6b0fcb (127.9.90.65:35601)
I20260619 10:25:57.608328 10560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772c024789054896b6916a6d4daf99f6" candidate_uuid: "4235da6d5b79426da92def935adf139a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9067360f2cb64f26b16b8b2ae95814ce"
I20260619 10:25:57.608358 10292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772c024789054896b6916a6d4daf99f6" candidate_uuid: "4235da6d5b79426da92def935adf139a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb"
I20260619 10:25:57.608403 10560 raft_consensus.cc:3037] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:57.608434 10292 raft_consensus.cc:3037] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:57.609133 10292 raft_consensus.cc:2454] T 772c024789054896b6916a6d4daf99f6 P bf4807ccdd454e42a16e37cc8b6b0fcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4235da6d5b79426da92def935adf139a in term 1.
I20260619 10:25:57.609153 10560 raft_consensus.cc:2454] T 772c024789054896b6916a6d4daf99f6 P 9067360f2cb64f26b16b8b2ae95814ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4235da6d5b79426da92def935adf139a in term 1.
I20260619 10:25:57.609295 10362 leader_election.cc:302] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4235da6d5b79426da92def935adf139a, bf4807ccdd454e42a16e37cc8b6b0fcb; no voters:
I20260619 10:25:57.609375 10627 raft_consensus.cc:2781] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:57.609529 10627 raft_consensus.cc:686] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [term 1 LEADER]: Becoming Leader. State: Replica: 4235da6d5b79426da92def935adf139a, State: Running, Role: LEADER
I20260619 10:25:57.609599 10627 consensus_queue.cc:237] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } }
I20260619 10:25:57.610410 10147 catalog_manager.cc:5495] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4235da6d5b79426da92def935adf139a (127.9.90.66). New cstate: current_term: 1 leader_uuid: "4235da6d5b79426da92def935adf139a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067360f2cb64f26b16b8b2ae95814ce" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4235da6d5b79426da92def935adf139a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf4807ccdd454e42a16e37cc8b6b0fcb" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35601 } health_report { overall_health: UNKNOWN } } }
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:25:57.619655 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.613465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.9.90.65:35601)
0619 10:25:57.613525 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.613527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.613581 (+ 54us) server_negotiation.cc:407] Connection header received
0619 10:25:57.613620 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.613621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.613628 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.613668 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:57.614164 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.614600 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.615214 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.615384 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.615426 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.617456 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:57.617465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:57.617465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:57.617471 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:57.618863 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.619142 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.619143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.619144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.619189 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.619307 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.619307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.619308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.619417 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:57.619516 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.619543 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.619594 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:25:57.640985 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.634322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.9.90.65:35601)
0619 10:25:57.634370 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.634372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.634468 (+ 96us) server_negotiation.cc:407] Connection header received
0619 10:25:57.634542 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.634542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.634552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.634593 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:57.635096 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.635567 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.636261 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.636443 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.636496 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.638506 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:57.638507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:57.638508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:57.638516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:57.639871 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.640242 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.640244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.640244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.640303 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.640472 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.640473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.640473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.640571 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:57.640671 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.640850 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.640918 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:25:57.661654 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:57.655002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.9.90.65:35601)
0619 10:25:57.655097 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:57.655100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:57.655184 (+ 84us) server_negotiation.cc:407] Connection header received
0619 10:25:57.655255 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:57.655257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:57.655268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:57.655325 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:57.655873 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.656378 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.657109 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:57.657293 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:57.657354 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:57.659499 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:57.659501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:57.659501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:57.659510 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:57.660593 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.660909 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.660910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.660910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.660965 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:57.661157 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:57.661158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:57.661159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:57.661256 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:57.661348 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:57.661482 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 10:25:57.661558 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 10:25:57.662107 10252 tablet_service.cc:2908] Rejecting scan request for tablet 772c024789054896b6916a6d4daf99f6: Uninitialized: clean time has not yet been initialized
I20260619 10:25:57.662935 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 10209
W20260619 10:25:57.667037 10473 tablet.cc:2369] T 772c024789054896b6916a6d4daf99f6 P 4235da6d5b79426da92def935adf139a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:25:57.667224 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 10343
I20260619 10:25:57.670993 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 10477
I20260619 10:25:57.674921 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 10115
2026-06-19T10:25:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1737 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10644](info): set up 2 sockets
Jun 19 10:25:57 dist-test-slave-hw7q krb5kdc[10644](info): commencing operation
krb5kdc: starting...
W20260619 10:25:58.625101 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:25:58 dist-test-slave-hw7q krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:25:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:25:58.729682 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:34409
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:34409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:58.806237 10657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:58.806442 10657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:58.806463 10657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:58.807560 10657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:25:58.807595 10657 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 10:25:58.807606 10657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:58.807617 10657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:58.807628 10657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:25:58.807638 10657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:25:58.809037 10657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:34409
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:34409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:58.809230 10657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:58.809406 10657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:25:58.813977 10657 server_base.cc:1034] running on GCE node
W20260619 10:25:58.814868 10665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:58.815178 10663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:58.815304 10662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:58.815552 10657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:58.815820 10657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:58.816962 10657 hybrid_clock.cc:648] HybridClock initialized: now 1781864758816949 us; error 31 us; skew 500 ppm
Jun 19 10:25:58 dist-test-slave-hw7q krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864758, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:58.819185 10657 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:25:58.819536 10657 webserver.cc:460] Webserver started at http://127.9.90.126:36765/ using document root <none> and password file <none>
I20260619 10:25:58.819700 10657 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:58.819736 10657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:58.819830 10657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:58.820783 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "edf526a026e3465ca1b563956a2a05a4"
format_stamp: "Formatted at 2026-06-19 10:25:58 on dist-test-slave-hw7q"
server_key: "58ccd140d32ed40e4b24916069f1c9c3"
server_key_iv: "a57d43926a50ff0486a3003955f3f5e0"
server_key_version: "encryptionkey@0"
I20260619 10:25:58.821061 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "edf526a026e3465ca1b563956a2a05a4"
format_stamp: "Formatted at 2026-06-19 10:25:58 on dist-test-slave-hw7q"
server_key: "58ccd140d32ed40e4b24916069f1c9c3"
server_key_iv: "a57d43926a50ff0486a3003955f3f5e0"
server_key_version: "encryptionkey@0"
I20260619 10:25:58.822214 10657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:25:58.822840 10672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:58.823028 10657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:58.823117 10657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "edf526a026e3465ca1b563956a2a05a4"
format_stamp: "Formatted at 2026-06-19 10:25:58 on dist-test-slave-hw7q"
server_key: "58ccd140d32ed40e4b24916069f1c9c3"
server_key_iv: "a57d43926a50ff0486a3003955f3f5e0"
server_key_version: "encryptionkey@0"
I20260619 10:25:58.823180 10657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:58.835345 10657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:58.836843 10657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:58.836941 10657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:58.840941 10657 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:34409
I20260619 10:25:58.841015 10734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:34409 every 8 connection(s)
I20260619 10:25:58.841272 10657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:25:58.841912 10735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:58.843389 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 10657
I20260619 10:25:58.843452 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:25:58.843578 9577 external_mini_cluster.cc:1428] Setting key 72e6fb6af904fe24610ebb4a43dbe3e9
I20260619 10:25:58.844413 10735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4: Bootstrap starting.
I20260619 10:25:58.845064 10735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:58.845345 10735 log.cc:824] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:58.846037 10735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4: No bootstrap required, opened a new log
I20260619 10:25:58.848205 10735 raft_consensus.cc:348] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf526a026e3465ca1b563956a2a05a4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 34409 } }
I20260619 10:25:58.848408 10735 raft_consensus.cc:374] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:58.848448 10735 raft_consensus.cc:729] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf526a026e3465ca1b563956a2a05a4, State: Initialized, Role: FOLLOWER
I20260619 10:25:58.848577 10735 consensus_queue.cc:260] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf526a026e3465ca1b563956a2a05a4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 34409 } }
I20260619 10:25:58.848656 10735 raft_consensus.cc:388] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:25:58.848678 10735 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:25:58.848713 10735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:58.849229 10735 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf526a026e3465ca1b563956a2a05a4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 34409 } }
I20260619 10:25:58.849321 10735 leader_election.cc:302] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edf526a026e3465ca1b563956a2a05a4; no voters:
I20260619 10:25:58.849473 10735 leader_election.cc:288] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:25:58.849504 10740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:58.849612 10740 raft_consensus.cc:686] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [term 1 LEADER]: Becoming Leader. State: Replica: edf526a026e3465ca1b563956a2a05a4, State: Running, Role: LEADER
I20260619 10:25:58.849690 10740 consensus_queue.cc:237] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf526a026e3465ca1b563956a2a05a4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 34409 } }
I20260619 10:25:58.849727 10735 sys_catalog.cc:564] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:25:58.850056 10740 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edf526a026e3465ca1b563956a2a05a4. Latest consensus state: current_term: 1 leader_uuid: "edf526a026e3465ca1b563956a2a05a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf526a026e3465ca1b563956a2a05a4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 34409 } } }
I20260619 10:25:58.850121 10740 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:58.850629 10745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 10:25:58 dist-test-slave-hw7q krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:58.851248 10745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:25:58.851859 10742 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edf526a026e3465ca1b563956a2a05a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf526a026e3465ca1b563956a2a05a4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 34409 } } }
I20260619 10:25:58.851987 10742 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:25:58.852917 10745 catalog_manager.cc:1347] Generated new cluster ID: e24af9c6e4e14ecd8ef35ad3629b901d
I20260619 10:25:58.852967 10745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:25:58.854430 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:58.844073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.9.90.126:34409)
0619 10:25:58.844250 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:58.844260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:58.844292 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:25:58.844822 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:58.844831 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:58.844895 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:58.845040 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:58.845595 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:58.846937 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:58.847747 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:58.848356 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:58.848471 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:58.851253 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:58.851258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:58.851258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:58.851270 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:58.853033 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:58.853350 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:58.853353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:58.853354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:58.853442 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:58.853613 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:58.853614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:58.853615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:58.853905 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:58.854033 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:58.854140 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 10:25:58.854268 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260619 10:25:58.869640 10745 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:25:58.870209 10745 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:25:58.876717 10745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4: Generated new TSK 0
I20260619 10:25:58.876906 10745 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:25:58.906198 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:34409
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:58.988216 10761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:58.988412 10761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:58.988435 10761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:58.989497 10761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:58.989527 10761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:58.989545 10761 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:25:58.989574 10761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:25:58.990891 10761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.90.126:34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:58.991062 10761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:58.991222 10761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:58.995633 10767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:58.995707 10769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:58.995822 10761 server_base.cc:1034] running on GCE node
W20260619 10:25:58.995846 10766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:58.996053 10761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:58.996269 10761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:58.997427 10761 hybrid_clock.cc:648] HybridClock initialized: now 1781864758997415 us; error 26 us; skew 500 ppm
Jun 19 10:25:58 dist-test-slave-hw7q krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864758, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:58.999729 10761 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:25:59.000144 10761 webserver.cc:460] Webserver started at http://127.9.90.65:42965/ using document root <none> and password file <none>
I20260619 10:25:59.000367 10761 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:59.000414 10761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:59.000536 10761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:59.001502 10761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "b28dd69da5f045fa9f91af9ff6371dc7"
format_stamp: "Formatted at 2026-06-19 10:25:58 on dist-test-slave-hw7q"
server_key: "eabae5b8fb2fffebeebbd1b8ac9d3cbe"
server_key_iv: "0a2f7a15c2a6c6d9521faedbc21c3568"
server_key_version: "encryptionkey@0"
I20260619 10:25:59.001873 10761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "b28dd69da5f045fa9f91af9ff6371dc7"
format_stamp: "Formatted at 2026-06-19 10:25:58 on dist-test-slave-hw7q"
server_key: "eabae5b8fb2fffebeebbd1b8ac9d3cbe"
server_key_iv: "0a2f7a15c2a6c6d9521faedbc21c3568"
server_key_version: "encryptionkey@0"
I20260619 10:25:59.003067 10761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:59.003803 10776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.003970 10761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:59.004047 10761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "b28dd69da5f045fa9f91af9ff6371dc7"
format_stamp: "Formatted at 2026-06-19 10:25:58 on dist-test-slave-hw7q"
server_key: "eabae5b8fb2fffebeebbd1b8ac9d3cbe"
server_key_iv: "0a2f7a15c2a6c6d9521faedbc21c3568"
server_key_version: "encryptionkey@0"
I20260619 10:25:59.004114 10761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:59.016997 10761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:59.018798 10761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:59.018930 10761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:59.019438 10761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:59.019476 10761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.019505 10761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:59.019533 10761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864758, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:59.025988 10761 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:36793
I20260619 10:25:59.026131 10891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:36793 every 8 connection(s)
I20260619 10:25:59.026376 10761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:25:59.028167 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.019848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:41023 (local address 127.9.90.126:34409)
0619 10:25:59.019906 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.019909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.020111 (+ 202us) server_negotiation.cc:407] Connection header received
0619 10:25:59.020683 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.020684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.020697 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.020743 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.021367 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.022087 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.023062 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.023328 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.023387 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.025567 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.025568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.025568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.025576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.027104 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.027448 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.027448 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.027449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.027496 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.027670 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.027670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.027671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.027766 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.027825 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.028034 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.028101 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:25:59.028290 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.019846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:34409 (local address 127.9.90.65:41023)
0619 10:25:59.019989 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.020027 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.020559 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.020826 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.020831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.021092 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:59.021279 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.021284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.022170 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.022172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.022982 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.022985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.023415 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.023417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.023551 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.024028 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:59.024038 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:59.025477 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:59.027185 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.027188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.027189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.027390 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.027541 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.027541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.027541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.027626 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.027832 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:59.027835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:59.027943 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.028018 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.028130 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260619 10:25:59.028776 10892 heartbeater.cc:347] Connected to a master server at 127.9.90.126:34409
I20260619 10:25:59.028851 10892 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:59.029011 10892 heartbeater.cc:511] Master 127.9.90.126:34409 requested a full tablet report, sending...
I20260619 10:25:59.029498 10688 ts_manager.cc:194] Registered new tserver with Master: b28dd69da5f045fa9f91af9ff6371dc7 (127.9.90.65:36793)
I20260619 10:25:59.030180 10688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:41023
I20260619 10:25:59.030607 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 10761
I20260619 10:25:59.030685 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:25:59.030835 9577 external_mini_cluster.cc:1428] Setting key c090cf92d105d5c1c491fb9286b71694
I20260619 10:25:59.035001 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.029737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:52769 (local address 127.9.90.126:34409)
0619 10:25:59.029775 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.029776 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.029907 (+ 131us) server_negotiation.cc:407] Connection header received
0619 10:25:59.029951 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.029952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.029962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.030012 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.030387 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.030909 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.031796 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.031991 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.032041 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.032817 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.032819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.032819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.032825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.034049 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.034339 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.034340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.034340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.034391 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.034548 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.034549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.034549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.034647 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.034699 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.034838 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.034911 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:25:59.035013 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.029693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:34409 (local address 127.9.90.65:52769)
0619 10:25:59.029846 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.029856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.029909 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.030015 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.030016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.030178 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:59.030237 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.030238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.030981 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.030982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.031724 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.031726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.032045 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.032046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.032106 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.032472 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:59.032476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:59.032740 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:59.034118 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.034119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.034119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.034280 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.034443 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.034443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.034444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.034498 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.034698 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:59.034699 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:59.034739 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.034809 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.034911 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:25:59.051868 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:34409
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:59.128719 10901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:59.128921 10901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:59.128943 10901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:59.130138 10901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:59.130182 10901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:59.130201 10901 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:25:59.130219 10901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:25:59.131620 10901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.90.126:34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:59.131851 10901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:59.132061 10901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:59.136760 10909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:59.136765 10907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:59.136768 10906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:59.136940 10901 server_base.cc:1034] running on GCE node
I20260619 10:25:59.137122 10901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:59.137313 10901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:59.138481 10901 hybrid_clock.cc:648] HybridClock initialized: now 1781864759138465 us; error 29 us; skew 500 ppm
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:59.140812 10901 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:25:59.141206 10901 webserver.cc:460] Webserver started at http://127.9.90.66:35925/ using document root <none> and password file <none>
I20260619 10:25:59.141422 10901 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:59.141466 10901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:59.141575 10901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:59.142608 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "26dc50861ec847098c4e34b1fc0d9c09"
format_stamp: "Formatted at 2026-06-19 10:25:59 on dist-test-slave-hw7q"
server_key: "7fa6a67c9b01536da9efd9df899194b4"
server_key_iv: "90522ed6f8ea698ce72ae3e4f6cb7bda"
server_key_version: "encryptionkey@0"
I20260619 10:25:59.142921 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "26dc50861ec847098c4e34b1fc0d9c09"
format_stamp: "Formatted at 2026-06-19 10:25:59 on dist-test-slave-hw7q"
server_key: "7fa6a67c9b01536da9efd9df899194b4"
server_key_iv: "90522ed6f8ea698ce72ae3e4f6cb7bda"
server_key_version: "encryptionkey@0"
I20260619 10:25:59.144171 10901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:25:59.144816 10916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.144954 10901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:59.145025 10901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "26dc50861ec847098c4e34b1fc0d9c09"
format_stamp: "Formatted at 2026-06-19 10:25:59 on dist-test-slave-hw7q"
server_key: "7fa6a67c9b01536da9efd9df899194b4"
server_key_iv: "90522ed6f8ea698ce72ae3e4f6cb7bda"
server_key_version: "encryptionkey@0"
I20260619 10:25:59.145088 10901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:59.168124 10901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:59.170288 10901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:59.170428 10901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:59.170909 10901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:59.170953 10901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.170974 10901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:59.170984 10901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:59.177392 10901 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:36851
I20260619 10:25:59.177459 11031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:36851 every 8 connection(s)
I20260619 10:25:59.177742 10901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:25:59.179968 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.171394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:53753 (local address 127.9.90.126:34409)
0619 10:25:59.171461 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.171464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.171659 (+ 195us) server_negotiation.cc:407] Connection header received
0619 10:25:59.172138 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.172140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.172153 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.172222 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.173020 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.173832 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.174795 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.175095 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.175162 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.177626 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.177627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.177627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.177634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.178965 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.179292 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.179293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.179293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.179339 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.179509 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.179510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.179510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.179602 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.179645 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.179852 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.179905 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:25:59.180127 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.171374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:34409 (local address 127.9.90.66:53753)
0619 10:25:59.171532 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.171573 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.172047 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.172302 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.172306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.172627 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:59.172921 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.172927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.173920 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.173921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.174714 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.174716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.175180 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.175181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.175327 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.175869 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:59.175879 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:59.177532 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:59.179032 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.179037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.179038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.179237 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.179382 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.179382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.179383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.179469 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.179643 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:59.179646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:59.179750 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.179835 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.179950 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260619 10:25:59.180546 11032 heartbeater.cc:347] Connected to a master server at 127.9.90.126:34409
I20260619 10:25:59.180625 11032 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:59.180801 11032 heartbeater.cc:511] Master 127.9.90.126:34409 requested a full tablet report, sending...
I20260619 10:25:59.181181 10688 ts_manager.cc:194] Registered new tserver with Master: 26dc50861ec847098c4e34b1fc0d9c09 (127.9.90.66:36851)
I20260619 10:25:59.181633 10688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:53753
I20260619 10:25:59.186062 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 10901
I20260619 10:25:59.186175 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:25:59.186223 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.181598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:56361 (local address 127.9.90.126:34409)
0619 10:25:59.181635 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.181637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.181752 (+ 115us) server_negotiation.cc:407] Connection header received
0619 10:25:59.181856 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.181857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.181866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.181921 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.182230 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.182676 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.183275 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.183439 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.183478 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.184152 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.184153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.184154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.184159 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.185182 (+ 1023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.185491 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.185491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.185492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.185533 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.185676 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.185676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.185676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.185840 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.185944 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.186042 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.186122 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:25:59.186246 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.181546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:34409 (local address 127.9.90.66:56361)
0619 10:25:59.181697 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.181706 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.181762 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.181924 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.181924 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.182100 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:59.182166 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.182168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.182748 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.182749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.183220 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.183221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.183485 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.183485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.183535 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.183850 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:59.183854 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:59.184089 (+ 235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:59.185265 (+ 1176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.185265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.185266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.185429 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.185581 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.185582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.185582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.185636 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.185903 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:59.185903 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:59.185950 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.186015 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.186149 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260619 10:25:59.186359 9577 external_mini_cluster.cc:1428] Setting key 558c8c56b12b794783c5f3f5a3bbbe9e
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:25:59.208072 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:34409
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:25:59.284844 11041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:25:59.285046 11041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:25:59.285069 11041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:25:59.286194 11041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:25:59.286231 11041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:25:59.286250 11041 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:25:59.286265 11041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:25:59.287871 11041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.90.126:34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:25:59.288069 11041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:25:59.288259 11041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:25:59.292896 11047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:59.292907 11046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:25:59.292915 11049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:25:59.292960 11041 server_base.cc:1034] running on GCE node
I20260619 10:25:59.293251 11041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:25:59.293502 11041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:25:59.294633 11041 hybrid_clock.cc:648] HybridClock initialized: now 1781864759294625 us; error 26 us; skew 500 ppm
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:25:59.299376 11041 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:25:59.299785 11041 webserver.cc:460] Webserver started at http://127.9.90.67:41233/ using document root <none> and password file <none>
I20260619 10:25:59.299981 11041 fs_manager.cc:359] Metadata directory not provided
I20260619 10:25:59.300028 11041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:25:59.300149 11041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:25:59.301115 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d"
format_stamp: "Formatted at 2026-06-19 10:25:59 on dist-test-slave-hw7q"
server_key: "9e7a160d5db8a54ec5589122967fc4fd"
server_key_iv: "84404264b1ba02af830e41f1e96dc918"
server_key_version: "encryptionkey@0"
I20260619 10:25:59.301431 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d"
format_stamp: "Formatted at 2026-06-19 10:25:59 on dist-test-slave-hw7q"
server_key: "9e7a160d5db8a54ec5589122967fc4fd"
server_key_iv: "84404264b1ba02af830e41f1e96dc918"
server_key_version: "encryptionkey@0"
I20260619 10:25:59.302708 11041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:25:59.303426 11056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.303565 11041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:25:59.303642 11041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d"
format_stamp: "Formatted at 2026-06-19 10:25:59 on dist-test-slave-hw7q"
server_key: "9e7a160d5db8a54ec5589122967fc4fd"
server_key_iv: "84404264b1ba02af830e41f1e96dc918"
server_key_version: "encryptionkey@0"
I20260619 10:25:59.303697 11041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:25:59.316943 11041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:25:59.318696 11041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:25:59.318822 11041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:25:59.319329 11041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:25:59.319367 11041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.319396 11041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:25:59.319418 11041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:59.326085 11041 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:40737
I20260619 10:25:59.326146 11171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:40737 every 8 connection(s)
I20260619 10:25:59.326470 11041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:25:59.328240 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.319692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:35279 (local address 127.9.90.126:34409)
0619 10:25:59.319753 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.319755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.319958 (+ 203us) server_negotiation.cc:407] Connection header received
0619 10:25:59.320433 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.320435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.320446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.320525 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.321143 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.321907 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.322952 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.323184 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.323240 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.325540 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.325542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.325542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.325549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.326995 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.327381 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.327382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.327383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.327440 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.327644 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.327645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.327645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.327761 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.327814 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.328081 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.328155 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:25:59.328382 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.319659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:34409 (local address 127.9.90.67:35279)
0619 10:25:59.319833 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.319872 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.320342 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.320595 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.320598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.320870 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:59.321057 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.321062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.321992 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.321993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.322884 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.322887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.323267 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.323268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.323372 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.323861 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:59.323871 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:59.325459 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:59.327071 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.327076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.327078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.327319 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.327492 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.327493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.327493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.327593 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.327814 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:59.327819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:59.327945 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.328030 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.328174 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260619 10:25:59.328809 11172 heartbeater.cc:347] Connected to a master server at 127.9.90.126:34409
I20260619 10:25:59.328883 11172 heartbeater.cc:464] Registering TS with master...
I20260619 10:25:59.329098 11172 heartbeater.cc:511] Master 127.9.90.126:34409 requested a full tablet report, sending...
I20260619 10:25:59.329470 10688 ts_manager.cc:194] Registered new tserver with Master: ff45df6ea44a4d4c9d74d92c8f07b71d (127.9.90.67:40737)
I20260619 10:25:59.329895 10688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:35279
I20260619 10:25:59.332369 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 11041
I20260619 10:25:59.332465 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:25:59.332629 9577 external_mini_cluster.cc:1428] Setting key b4503c2777928f64ef72bb08bc55eed7
I20260619 10:25:59.333518 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:25:59.335017 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.329882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:48377 (local address 127.9.90.126:34409)
0619 10:25:59.329929 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.329931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.330018 (+ 87us) server_negotiation.cc:407] Connection header received
0619 10:25:59.330070 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.330071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.330081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.330125 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.330407 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.330853 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.331523 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.331717 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.331769 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.332711 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.332712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.332713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.332720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.334066 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.334355 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.334356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.334356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.334406 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.334560 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.334561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.334562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.334679 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.334733 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.334864 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.334929 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:25:59.335031 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.329814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:34409 (local address 127.9.90.67:48377)
0619 10:25:59.329961 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.329972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.330027 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.330128 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.330128 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.330284 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:59.330344 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.330345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.330932 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.330933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.331454 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.331456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.331773 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.331774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.331846 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.332261 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:59.332266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:59.332603 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:59.334142 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.334143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.334144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.334296 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.334456 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.334457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.334457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.334513 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.334733 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:59.334733 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:59.334777 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.334837 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.334936 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":55,"threads_started":1}
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:59.353996 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.347724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.9.90.126:34409)
0619 10:25:59.347767 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.347769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.347777 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:25:59.347835 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.347836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.347843 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.347893 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.348350 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.348887 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.349699 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.349898 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.349947 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.352151 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.352152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.352153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.352159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.353171 (+ 1012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.353449 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.353449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.353450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.353491 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.353610 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.353610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.353610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.353693 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.353755 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.353883 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.353936 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:25:59.355373 10688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:25:59.356824 10688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:59.362486 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.358867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36793 (local address 127.0.0.1:57210)
0619 10:25:59.359109 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.359119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.359170 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.359330 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.359330 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.359332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:59.359376 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.359377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.360245 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.360246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.361447 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.361449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.362187 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.362188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.362281 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.362295 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.362357 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.362413 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":42,"threads_started":1}
I20260619 10:25:59.362501 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.359035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.9.90.65:36793)
0619 10:25:59.359195 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.359200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.359209 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:59.359245 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.359248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.359260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.359333 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:59.359425 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.360174 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.361512 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.362114 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.362179 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.362240 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.362376 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.362416 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260619 10:25:59.362998 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.358685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:36851 (local address 127.0.0.1:40906)
0619 10:25:59.358822 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.358862 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.358925 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.359225 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.359225 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.359227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:59.359298 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.359300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.360583 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.360583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.361605 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.361607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.362829 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.362829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.362892 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.362895 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.362919 (+ 24us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.362950 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":7,"thread_start_us":51,"threads_started":1}
I20260619 10:25:59.363200 10826 tablet_service.cc:1467] Processing CreateTablet for tablet ea6cc268b183474b8a5b63e9d2253af1 (DEFAULT_TABLE table=test-table [id=cb3920ac398246cfab229b1a492e3821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:59.363351 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.358948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.9.90.66:36851)
0619 10:25:59.359085 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.359088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.359096 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:25:59.359126 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.359128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.359139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.359374 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:59.359431 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.360503 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.361690 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.362461 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.363149 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.363219 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.363243 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.363277 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260619 10:25:59.363584 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6cc268b183474b8a5b63e9d2253af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:59.363865 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.358678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:40737 (local address 127.0.0.1:59296)
0619 10:25:59.358822 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.358862 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.358925 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.359221 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.359224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.359226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:59.359300 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.359301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.361685 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.361685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.362769 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.362772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.363646 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.363647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.363710 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.363713 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.363764 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.363804 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260619 10:25:59.364068 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.358766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.9.90.67:40737)
0619 10:25:59.359055 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.359060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.359070 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:25:59.359107 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.359109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.359124 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.359374 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:59.359431 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.360503 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.362831 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.363527 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.363628 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.363686 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.363797 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.363852 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":87,"threads_started":1}
I20260619 10:25:59.364590 11106 tablet_service.cc:1467] Processing CreateTablet for tablet ea6cc268b183474b8a5b63e9d2253af1 (DEFAULT_TABLE table=test-table [id=cb3920ac398246cfab229b1a492e3821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:59.364863 11106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6cc268b183474b8a5b63e9d2253af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:59.366560 11192 tablet_bootstrap.cc:492] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: Bootstrap starting.
I20260619 10:25:59.366904 10966 tablet_service.cc:1467] Processing CreateTablet for tablet ea6cc268b183474b8a5b63e9d2253af1 (DEFAULT_TABLE table=test-table [id=cb3920ac398246cfab229b1a492e3821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:59.367165 10966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6cc268b183474b8a5b63e9d2253af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:59.367192 11192 tablet_bootstrap.cc:654] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:59.367437 11193 tablet_bootstrap.cc:492] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: Bootstrap starting.
I20260619 10:25:59.367461 11192 log.cc:824] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:59.368063 11192 tablet_bootstrap.cc:492] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: No bootstrap required, opened a new log
I20260619 10:25:59.368122 11192 ts_tablet_manager.cc:1397] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:59.368309 11193 tablet_bootstrap.cc:654] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:59.368637 11193 log.cc:824] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:59.369336 11193 tablet_bootstrap.cc:492] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: No bootstrap required, opened a new log
I20260619 10:25:59.369401 11193 ts_tablet_manager.cc:1397] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:59.369680 11192 raft_consensus.cc:348] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } }
I20260619 10:25:59.369844 11192 raft_consensus.cc:374] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:59.369885 11192 raft_consensus.cc:729] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28dd69da5f045fa9f91af9ff6371dc7, State: Initialized, Role: FOLLOWER
I20260619 10:25:59.369992 11192 consensus_queue.cc:260] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } }
I20260619 10:25:59.370223 11192 ts_tablet_manager.cc:1428] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:59.370338 10892 heartbeater.cc:503] Master 127.9.90.126:34409 was elected leader, sending a full tablet report...
I20260619 10:25:59.370473 11197 tablet_bootstrap.cc:492] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: Bootstrap starting.
I20260619 10:25:59.371028 11197 tablet_bootstrap.cc:654] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:59.371320 11197 log.cc:824] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: Log is configured to *not* fsync() on all Append() calls
I20260619 10:25:59.371312 11193 raft_consensus.cc:348] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } }
I20260619 10:25:59.371438 11193 raft_consensus.cc:374] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:59.371474 11193 raft_consensus.cc:729] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff45df6ea44a4d4c9d74d92c8f07b71d, State: Initialized, Role: FOLLOWER
I20260619 10:25:59.371582 11193 consensus_queue.cc:260] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } }
I20260619 10:25:59.371809 11193 ts_tablet_manager.cc:1428] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:25:59.371860 11172 heartbeater.cc:503] Master 127.9.90.126:34409 was elected leader, sending a full tablet report...
I20260619 10:25:59.372344 11197 tablet_bootstrap.cc:492] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: No bootstrap required, opened a new log
I20260619 10:25:59.372398 11197 ts_tablet_manager.cc:1397] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:25:59.373698 11197 raft_consensus.cc:348] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } }
I20260619 10:25:59.373884 11197 raft_consensus.cc:374] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:59.373921 11197 raft_consensus.cc:729] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26dc50861ec847098c4e34b1fc0d9c09, State: Initialized, Role: FOLLOWER
I20260619 10:25:59.374027 11197 consensus_queue.cc:260] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } }
I20260619 10:25:59.374274 11197 ts_tablet_manager.cc:1428] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:25:59.374287 11032 heartbeater.cc:503] Master 127.9.90.126:34409 was elected leader, sending a full tablet report...
W20260619 10:25:59.428267 11033 tablet.cc:2369] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:25:59.527078 10893 tablet.cc:2369] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:25:59.561854 11200 raft_consensus.cc:482] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:59.561977 11200 raft_consensus.cc:504] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } }
I20260619 10:25:59.562346 11200 leader_election.cc:288] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff45df6ea44a4d4c9d74d92c8f07b71d (127.9.90.67:40737), b28dd69da5f045fa9f91af9ff6371dc7 (127.9.90.65:36793)
I20260619 10:25:59.565169 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.562648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:58923 (local address 127.9.90.67:40737)
0619 10:25:59.562706 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.562708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.562716 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:25:59.562746 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.562746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.562758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.562858 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:59.563005 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.563560 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.564506 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.564841 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.564892 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.564900 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.565065 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.565100 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:25:59.565186 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.562892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:51379 (local address 127.9.90.65:36793)
0619 10:25:59.562943 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.562945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.562953 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:25:59.562976 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.562977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.562988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.563039 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:59.563191 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.563662 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.564532 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.564877 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.564944 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.564954 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.565088 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.565122 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:25:59.565186 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.562529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:40737 (local address 127.9.90.66:58923)
0619 10:25:59.562594 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.562607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.562667 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.562868 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.562870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.562872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:59.562941 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.562943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.563639 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.563639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.564415 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.564417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.564914 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.564915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.564997 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.565001 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.565042 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.565076 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 10:25:59.565196 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.562694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36793 (local address 127.9.90.66:51379)
0619 10:25:59.562850 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.562860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.562912 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.563059 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.563060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.563062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:59.563130 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.563132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.563741 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.563742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.564441 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.564443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.564958 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.564959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.565018 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.565021 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.565071 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.565105 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260619 10:25:59.565555 11126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6cc268b183474b8a5b63e9d2253af1" candidate_uuid: "26dc50861ec847098c4e34b1fc0d9c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" is_pre_election: true
I20260619 10:25:59.565716 11126 raft_consensus.cc:2454] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26dc50861ec847098c4e34b1fc0d9c09 in term 0.
I20260619 10:25:59.565827 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6cc268b183474b8a5b63e9d2253af1" candidate_uuid: "26dc50861ec847098c4e34b1fc0d9c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" is_pre_election: true
I20260619 10:25:59.565977 10841 raft_consensus.cc:2454] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26dc50861ec847098c4e34b1fc0d9c09 in term 0.
I20260619 10:25:59.565977 10919 leader_election.cc:302] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26dc50861ec847098c4e34b1fc0d9c09, ff45df6ea44a4d4c9d74d92c8f07b71d; no voters:
I20260619 10:25:59.566088 11200 raft_consensus.cc:2781] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:59.566149 11200 raft_consensus.cc:482] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:59.566166 11200 raft_consensus.cc:3037] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:59.567041 11200 raft_consensus.cc:504] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } }
I20260619 10:25:59.567162 11200 leader_election.cc:288] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [CANDIDATE]: Term 1 election: Requested vote from peers ff45df6ea44a4d4c9d74d92c8f07b71d (127.9.90.67:40737), b28dd69da5f045fa9f91af9ff6371dc7 (127.9.90.65:36793)
I20260619 10:25:59.567349 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6cc268b183474b8a5b63e9d2253af1" candidate_uuid: "26dc50861ec847098c4e34b1fc0d9c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28dd69da5f045fa9f91af9ff6371dc7"
I20260619 10:25:59.567342 11126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6cc268b183474b8a5b63e9d2253af1" candidate_uuid: "26dc50861ec847098c4e34b1fc0d9c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d"
I20260619 10:25:59.567415 10841 raft_consensus.cc:3037] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:59.567416 11126 raft_consensus.cc:3037] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:59.568023 10841 raft_consensus.cc:2454] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26dc50861ec847098c4e34b1fc0d9c09 in term 1.
I20260619 10:25:59.568133 11126 raft_consensus.cc:2454] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26dc50861ec847098c4e34b1fc0d9c09 in term 1.
I20260619 10:25:59.568169 10919 leader_election.cc:302] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26dc50861ec847098c4e34b1fc0d9c09, b28dd69da5f045fa9f91af9ff6371dc7; no voters:
I20260619 10:25:59.568277 11200 raft_consensus.cc:2781] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:59.568425 11200 raft_consensus.cc:686] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 LEADER]: Becoming Leader. State: Replica: 26dc50861ec847098c4e34b1fc0d9c09, State: Running, Role: LEADER
I20260619 10:25:59.568519 11200 consensus_queue.cc:237] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } }
I20260619 10:25:59.569278 10688 catalog_manager.cc:5495] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26dc50861ec847098c4e34b1fc0d9c09 (127.9.90.66). New cstate: current_term: 1 leader_uuid: "26dc50861ec847098c4e34b1fc0d9c09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } health_report { overall_health: HEALTHY } } }
W20260619 10:25:59.576983 11173 tablet.cc:2369] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:25:59.591343 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.589091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.9.90.66:36851)
0619 10:25:59.589139 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.589141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.589151 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:25:59.589233 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.589234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.589250 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.589312 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:59.589472 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.589972 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.590718 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.590894 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.590934 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.591078 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:59.591163 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:59.591212 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.591244 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.591260 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 10:25:59.592913 10841 raft_consensus.cc:1261] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Refusing update from remote peer 26dc50861ec847098c4e34b1fc0d9c09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:59.592935 11126 raft_consensus.cc:1261] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Refusing update from remote peer 26dc50861ec847098c4e34b1fc0d9c09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:59.593215 11200 consensus_queue.cc:1035] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:59.593344 11202 consensus_queue.cc:1035] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:59.595516 11205 mvcc.cc:205] Tried to move back new op lower bound from 7298518055290159104 to 7298518055193116672. Current Snapshot: MvccSnapshot[applied={T|T < 7298518055290159104}]
I20260619 10:25:59.596199 11206 mvcc.cc:205] Tried to move back new op lower bound from 7298518055290159104 to 7298518055193116672. Current Snapshot: MvccSnapshot[applied={T|T < 7298518055290159104}]
I20260619 10:25:59.598738 10688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:59.598837 10688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:59.599537 10688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4: Sending DeleteTablet for 3 replicas of tablet ea6cc268b183474b8a5b63e9d2253af1
I20260619 10:25:59.599844 10966 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6cc268b183474b8a5b63e9d2253af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:59 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:40906
I20260619 10:25:59.599879 10826 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6cc268b183474b8a5b63e9d2253af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:59 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:57210
I20260619 10:25:59.600039 11106 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6cc268b183474b8a5b63e9d2253af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:59 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:59296
I20260619 10:25:59.600128 10688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 10:25:59.600231 11215 tablet_replica.cc:330] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: stopping tablet replica
W20260619 10:25:59.600267 10688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:59.600317 11215 raft_consensus.cc:2229] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:59.600410 11215 raft_consensus.cc:2258] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:59.600741 11216 tablet_replica.cc:330] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: stopping tablet replica
I20260619 10:25:59.600826 11215 ts_tablet_manager.cc:1905] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:59.600930 11216 raft_consensus.cc:2229] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:59.601081 11216 raft_consensus.cc:2258] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:59.601429 10966 tablet_service.cc:1467] Processing CreateTablet for tablet f7d4d474356a4e9796a6c14e40c6c28a (DEFAULT_TABLE table=test-table [id=91a88c3d6c03413381c6289cfe863978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:59.601547 10966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d4d474356a4e9796a6c14e40c6c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:59.601671 11216 ts_tablet_manager.cc:1905] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:59.602388 11215 ts_tablet_manager.cc:1918] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:59.602458 11215 log.cc:1196] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/ea6cc268b183474b8a5b63e9d2253af1
I20260619 10:25:59.602674 11215 ts_tablet_manager.cc:1939] T ea6cc268b183474b8a5b63e9d2253af1 P b28dd69da5f045fa9f91af9ff6371dc7: Deleting consensus metadata
I20260619 10:25:59.603487 11216 ts_tablet_manager.cc:1918] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:59.603621 11216 log.cc:1196] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/ea6cc268b183474b8a5b63e9d2253af1
I20260619 10:25:59.603891 11216 ts_tablet_manager.cc:1939] T ea6cc268b183474b8a5b63e9d2253af1 P ff45df6ea44a4d4c9d74d92c8f07b71d: Deleting consensus metadata
I20260619 10:25:59.603894 10675 catalog_manager.cc:4841] TS b28dd69da5f045fa9f91af9ff6371dc7 (127.9.90.65:36793): tablet ea6cc268b183474b8a5b63e9d2253af1 (table test-table [id=cb3920ac398246cfab229b1a492e3821]) successfully deleted
I20260619 10:25:59.603997 10826 tablet_service.cc:1467] Processing CreateTablet for tablet f7d4d474356a4e9796a6c14e40c6c28a (DEFAULT_TABLE table=test-table [id=91a88c3d6c03413381c6289cfe863978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:59.604099 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d4d474356a4e9796a6c14e40c6c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:59.604557 11197 tablet_bootstrap.cc:492] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: Bootstrap starting.
I20260619 10:25:59.604939 11197 tablet_bootstrap.cc:654] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:59.605212 11192 tablet_bootstrap.cc:492] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: Bootstrap starting.
I20260619 10:25:59.605254 11106 tablet_service.cc:1467] Processing CreateTablet for tablet f7d4d474356a4e9796a6c14e40c6c28a (DEFAULT_TABLE table=test-table [id=91a88c3d6c03413381c6289cfe863978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:25:59.605340 11106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d4d474356a4e9796a6c14e40c6c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:59.605564 11192 tablet_bootstrap.cc:654] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:59.605648 11197 tablet_bootstrap.cc:492] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: No bootstrap required, opened a new log
I20260619 10:25:59.605686 11197 ts_tablet_manager.cc:1397] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:59.605837 11197 raft_consensus.cc:348] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.605914 11197 raft_consensus.cc:374] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:59.605939 11197 raft_consensus.cc:729] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26dc50861ec847098c4e34b1fc0d9c09, State: Initialized, Role: FOLLOWER
I20260619 10:25:59.605990 11197 consensus_queue.cc:260] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.606084 11197 ts_tablet_manager.cc:1428] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.606604 11193 tablet_bootstrap.cc:492] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: Bootstrap starting.
I20260619 10:25:59.606613 11214 tablet_replica.cc:330] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: stopping tablet replica
I20260619 10:25:59.606765 11214 raft_consensus.cc:2229] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:25:59.606890 11214 raft_consensus.cc:2258] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:59.606990 11193 tablet_bootstrap.cc:654] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:59.607311 11214 ts_tablet_manager.cc:1905] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:59.607357 10675 catalog_manager.cc:4841] TS ff45df6ea44a4d4c9d74d92c8f07b71d (127.9.90.67:40737): tablet ea6cc268b183474b8a5b63e9d2253af1 (table test-table [id=cb3920ac398246cfab229b1a492e3821]) successfully deleted
I20260619 10:25:59.607875 11193 tablet_bootstrap.cc:492] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: No bootstrap required, opened a new log
I20260619 10:25:59.607924 11193 ts_tablet_manager.cc:1397] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:59.608060 11193 raft_consensus.cc:348] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.608122 11193 raft_consensus.cc:374] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:59.608141 11193 raft_consensus.cc:729] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff45df6ea44a4d4c9d74d92c8f07b71d, State: Initialized, Role: FOLLOWER
I20260619 10:25:59.608184 11193 consensus_queue.cc:260] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.608280 11193 ts_tablet_manager.cc:1428] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:59.608815 11192 tablet_bootstrap.cc:492] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: No bootstrap required, opened a new log
I20260619 10:25:59.608861 11192 ts_tablet_manager.cc:1397] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260619 10:25:59.609000 11192 raft_consensus.cc:348] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.609066 11192 raft_consensus.cc:374] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:59.609081 11192 raft_consensus.cc:729] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28dd69da5f045fa9f91af9ff6371dc7, State: Initialized, Role: FOLLOWER
I20260619 10:25:59.609118 11192 consensus_queue.cc:260] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.609210 11192 ts_tablet_manager.cc:1428] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.609764 11214 ts_tablet_manager.cc:1918] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:25:59.609843 11214 log.cc:1196] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/ea6cc268b183474b8a5b63e9d2253af1
I20260619 10:25:59.610010 11214 ts_tablet_manager.cc:1939] T ea6cc268b183474b8a5b63e9d2253af1 P 26dc50861ec847098c4e34b1fc0d9c09: Deleting consensus metadata
I20260619 10:25:59.610287 10676 catalog_manager.cc:4841] TS 26dc50861ec847098c4e34b1fc0d9c09 (127.9.90.66:36851): tablet ea6cc268b183474b8a5b63e9d2253af1 (table test-table [id=cb3920ac398246cfab229b1a492e3821]) successfully deleted
I20260619 10:25:59.681382 11202 raft_consensus.cc:482] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:59.681471 11202 raft_consensus.cc:504] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.681619 11202 leader_election.cc:288] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b28dd69da5f045fa9f91af9ff6371dc7 (127.9.90.65:36793), ff45df6ea44a4d4c9d74d92c8f07b71d (127.9.90.67:40737)
I20260619 10:25:59.681860 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d4d474356a4e9796a6c14e40c6c28a" candidate_uuid: "26dc50861ec847098c4e34b1fc0d9c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" is_pre_election: true
I20260619 10:25:59.681906 11126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d4d474356a4e9796a6c14e40c6c28a" candidate_uuid: "26dc50861ec847098c4e34b1fc0d9c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" is_pre_election: true
I20260619 10:25:59.681962 10841 raft_consensus.cc:2454] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26dc50861ec847098c4e34b1fc0d9c09 in term 0.
I20260619 10:25:59.681979 11126 raft_consensus.cc:2454] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26dc50861ec847098c4e34b1fc0d9c09 in term 0.
I20260619 10:25:59.682106 10919 leader_election.cc:302] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26dc50861ec847098c4e34b1fc0d9c09, b28dd69da5f045fa9f91af9ff6371dc7; no voters:
I20260619 10:25:59.682199 11202 raft_consensus.cc:2781] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:59.682237 11202 raft_consensus.cc:482] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:59.682255 11202 raft_consensus.cc:3037] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:59.683055 11202 raft_consensus.cc:504] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.683173 11202 leader_election.cc:288] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [CANDIDATE]: Term 1 election: Requested vote from peers b28dd69da5f045fa9f91af9ff6371dc7 (127.9.90.65:36793), ff45df6ea44a4d4c9d74d92c8f07b71d (127.9.90.67:40737)
I20260619 10:25:59.683373 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d4d474356a4e9796a6c14e40c6c28a" candidate_uuid: "26dc50861ec847098c4e34b1fc0d9c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28dd69da5f045fa9f91af9ff6371dc7"
I20260619 10:25:59.683451 10841 raft_consensus.cc:3037] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:59.683405 11126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d4d474356a4e9796a6c14e40c6c28a" candidate_uuid: "26dc50861ec847098c4e34b1fc0d9c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d"
I20260619 10:25:59.683485 11126 raft_consensus.cc:3037] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:59.684301 10841 raft_consensus.cc:2454] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26dc50861ec847098c4e34b1fc0d9c09 in term 1.
I20260619 10:25:59.684299 11126 raft_consensus.cc:2454] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26dc50861ec847098c4e34b1fc0d9c09 in term 1.
I20260619 10:25:59.684458 10919 leader_election.cc:302] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26dc50861ec847098c4e34b1fc0d9c09, b28dd69da5f045fa9f91af9ff6371dc7; no voters:
I20260619 10:25:59.684572 11202 raft_consensus.cc:2781] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:59.684618 11202 raft_consensus.cc:686] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 LEADER]: Becoming Leader. State: Replica: 26dc50861ec847098c4e34b1fc0d9c09, State: Running, Role: LEADER
I20260619 10:25:59.684674 11202 consensus_queue.cc:237] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.685256 10688 catalog_manager.cc:5495] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26dc50861ec847098c4e34b1fc0d9c09 (127.9.90.66). New cstate: current_term: 1 leader_uuid: "26dc50861ec847098c4e34b1fc0d9c09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } health_report { overall_health: UNKNOWN } } }
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864758, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:25:59.733318 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.726471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:34409 (local address 127.0.0.1:46908)
0619 10:25:59.726620 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.726631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.726696 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.726811 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.726812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.727159 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:25:59.727214 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.727216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.728083 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.728084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.728721 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.728723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.729063 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.729064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.729115 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.729556 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:25:59.729560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:25:59.730956 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:25:59.732339 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.732347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.732349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.732545 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.732746 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:25:59.732747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:25:59.732747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:25:59.732798 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:25:59.733041 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:25:59.733045 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:25:59.733088 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.733130 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.733244 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260619 10:25:59.733398 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.726510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.9.90.126:34409)
0619 10:25:59.726560 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.726562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.726688 (+ 126us) server_negotiation.cc:407] Connection header received
0619 10:25:59.726740 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.726741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.726751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.726801 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.727280 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.728020 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.728780 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.729007 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.729063 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.731061 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.731062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.731063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.731070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.732256 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.732623 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.732625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.732625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.732689 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.732870 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.732871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.732872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.732985 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.733047 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.733183 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.733296 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":26}
I20260619 10:25:59.735174 10688 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46908:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 10:25:59.735364 10688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:25:59.736644 10826 tablet_service.cc:1467] Processing CreateTablet for tablet d02f8e5068fb4dd68e87050d7d8f3d6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1096c44f171f45d0806ed489d2416333]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:25:59.736696 11106 tablet_service.cc:1467] Processing CreateTablet for tablet d02f8e5068fb4dd68e87050d7d8f3d6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1096c44f171f45d0806ed489d2416333]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:25:59.736706 10966 tablet_service.cc:1467] Processing CreateTablet for tablet d02f8e5068fb4dd68e87050d7d8f3d6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1096c44f171f45d0806ed489d2416333]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:25:59.736811 11106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02f8e5068fb4dd68e87050d7d8f3d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:59.736810 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02f8e5068fb4dd68e87050d7d8f3d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:59.736819 10966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02f8e5068fb4dd68e87050d7d8f3d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:25:59.738266 11193 tablet_bootstrap.cc:492] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: Bootstrap starting.
I20260619 10:25:59.738574 11197 tablet_bootstrap.cc:492] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: Bootstrap starting.
I20260619 10:25:59.738682 11193 tablet_bootstrap.cc:654] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:59.738903 11197 tablet_bootstrap.cc:654] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:59.739605 11197 tablet_bootstrap.cc:492] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: No bootstrap required, opened a new log
I20260619 10:25:59.739647 11197 ts_tablet_manager.cc:1397] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:25:59.739756 11197 raft_consensus.cc:348] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.739815 11197 raft_consensus.cc:374] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:59.739828 11197 raft_consensus.cc:729] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26dc50861ec847098c4e34b1fc0d9c09, State: Initialized, Role: FOLLOWER
I20260619 10:25:59.739873 11197 consensus_queue.cc:260] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.739962 11197 ts_tablet_manager.cc:1428] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.740018 11200 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.740118 11200 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:25:59.740782 11192 tablet_bootstrap.cc:492] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: Bootstrap starting.
I20260619 10:25:59.741087 11193 tablet_bootstrap.cc:492] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: No bootstrap required, opened a new log
I20260619 10:25:59.741120 11193 ts_tablet_manager.cc:1397] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 10:25:59.741123 11192 tablet_bootstrap.cc:654] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: Neither blocks nor log segments found. Creating new log.
I20260619 10:25:59.741286 11193 raft_consensus.cc:348] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.741369 11193 raft_consensus.cc:374] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:59.741387 11193 raft_consensus.cc:729] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff45df6ea44a4d4c9d74d92c8f07b71d, State: Initialized, Role: FOLLOWER
I20260619 10:25:59.741427 11193 consensus_queue.cc:260] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.741515 11193 ts_tablet_manager.cc:1428] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:25:59.741564 11199 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.741670 11199 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:25:59.741888 11192 tablet_bootstrap.cc:492] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: No bootstrap required, opened a new log
I20260619 10:25:59.741931 11192 ts_tablet_manager.cc:1397] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:25:59.742059 11192 raft_consensus.cc:348] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.742111 11192 raft_consensus.cc:374] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:25:59.742122 11192 raft_consensus.cc:729] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28dd69da5f045fa9f91af9ff6371dc7, State: Initialized, Role: FOLLOWER
I20260619 10:25:59.742162 11192 consensus_queue.cc:260] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.742259 11192 ts_tablet_manager.cc:1428] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:25:59.742326 11196 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.742425 11196 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:25:59.833925 11199 raft_consensus.cc:482] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:25:59.834021 11199 raft_consensus.cc:504] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.834334 11199 leader_election.cc:288] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b28dd69da5f045fa9f91af9ff6371dc7 (127.9.90.65:36793), 26dc50861ec847098c4e34b1fc0d9c09 (127.9.90.66:36851)
I20260619 10:25:59.838124 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.834589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:45909 (local address 127.9.90.66:36851)
0619 10:25:59.834683 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.834684 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.834693 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:59.834747 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.834748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.834756 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.834815 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:59.834999 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.835753 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.837179 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.837756 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.837849 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.837860 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.838020 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.838068 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 10:25:59.838133 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.834585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:52803 (local address 127.9.90.65:36793)
0619 10:25:59.834640 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.834642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.834691 (+ 49us) server_negotiation.cc:407] Connection header received
0619 10:25:59.834746 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.834747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.834757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.834815 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:25:59.835003 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.835746 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.837208 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.837739 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.837851 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.837860 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.838042 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.838082 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:25:59.838327 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.834466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36793 (local address 127.9.90.67:52803)
0619 10:25:59.834616 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.834627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.834694 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.834825 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.834826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.834828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:59.834900 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.834902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.835828 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.835830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.837079 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.837083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.837889 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.837891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.837957 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.837960 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.838211 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.838263 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":24,"thread_start_us":78,"threads_started":1}
I20260619 10:25:59.838351 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.834475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:36851 (local address 127.9.90.67:45909)
0619 10:25:59.834642 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.834651 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.834694 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.834829 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.834830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.834831 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:25:59.834901 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.834902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.835835 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.835835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.837080 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.837083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.837852 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.837853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.837933 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.837938 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.838205 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.838257 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":52,"thread_start_us":65,"threads_started":1}
I20260619 10:25:59.838589 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02f8e5068fb4dd68e87050d7d8f3d6a" candidate_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" is_pre_election: true
I20260619 10:25:59.838577 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02f8e5068fb4dd68e87050d7d8f3d6a" candidate_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26dc50861ec847098c4e34b1fc0d9c09" is_pre_election: true
I20260619 10:25:59.838678 10841 raft_consensus.cc:2454] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff45df6ea44a4d4c9d74d92c8f07b71d in term 0.
I20260619 10:25:59.838696 10986 raft_consensus.cc:2454] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff45df6ea44a4d4c9d74d92c8f07b71d in term 0.
I20260619 10:25:59.838840 11059 leader_election.cc:302] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b28dd69da5f045fa9f91af9ff6371dc7, ff45df6ea44a4d4c9d74d92c8f07b71d; no voters:
I20260619 10:25:59.838986 11199 raft_consensus.cc:2781] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:25:59.839033 11199 raft_consensus.cc:482] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:25:59.839061 11199 raft_consensus.cc:3037] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:59.839747 11199 raft_consensus.cc:504] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.839869 11199 leader_election.cc:288] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [CANDIDATE]: Term 1 election: Requested vote from peers b28dd69da5f045fa9f91af9ff6371dc7 (127.9.90.65:36793), 26dc50861ec847098c4e34b1fc0d9c09 (127.9.90.66:36851)
I20260619 10:25:59.840018 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02f8e5068fb4dd68e87050d7d8f3d6a" candidate_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26dc50861ec847098c4e34b1fc0d9c09"
I20260619 10:25:59.840026 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02f8e5068fb4dd68e87050d7d8f3d6a" candidate_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28dd69da5f045fa9f91af9ff6371dc7"
I20260619 10:25:59.840097 10841 raft_consensus.cc:3037] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:59.840097 10986 raft_consensus.cc:3037] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:25:59.840822 10841 raft_consensus.cc:2454] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff45df6ea44a4d4c9d74d92c8f07b71d in term 1.
I20260619 10:25:59.840816 10986 raft_consensus.cc:2454] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff45df6ea44a4d4c9d74d92c8f07b71d in term 1.
I20260619 10:25:59.841006 11060 leader_election.cc:302] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26dc50861ec847098c4e34b1fc0d9c09, ff45df6ea44a4d4c9d74d92c8f07b71d; no voters:
I20260619 10:25:59.841094 11199 raft_consensus.cc:2781] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:25:59.841178 11199 raft_consensus.cc:686] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 LEADER]: Becoming Leader. State: Replica: ff45df6ea44a4d4c9d74d92c8f07b71d, State: Running, Role: LEADER
I20260619 10:25:59.841264 11199 consensus_queue.cc:237] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } }
I20260619 10:25:59.841521 11199 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: TxnStatusTablet state changed. Reason: New leader ff45df6ea44a4d4c9d74d92c8f07b71d. Latest consensus state: current_term: 1 leader_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.841588 11199 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:25:59.841820 11234 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 10:25:59.841941 11234 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 10:25:59.841943 10688 catalog_manager.cc:5495] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d reported cstate change: term changed from 0 to 1, leader changed from <none> to ff45df6ea44a4d4c9d74d92c8f07b71d (127.9.90.67). New cstate: current_term: 1 leader_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } health_report { overall_health: HEALTHY } } }
I20260619 10:25:59.856829 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.854339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.9.90.67:40737)
0619 10:25:59.854376 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.854378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.854403 (+ 25us) server_negotiation.cc:407] Connection header received
0619 10:25:59.854499 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.854499 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.854513 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.854566 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:59.854696 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.855138 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.856164 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.856333 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.856371 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.856560 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:59.856643 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:59.856689 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.856759 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.856775 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:25:59.856884 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.854276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:40737 (local address 127.0.0.1:59302)
0619 10:25:59.854358 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.854368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.854422 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.854573 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.854574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.854575 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:25:59.854634 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.854637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.855199 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.855200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.856078 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.856089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.856390 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.856391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.856458 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.856474 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:25:59.856696 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:25:59.856701 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.856759 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.856804 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 10:25:59.857779 10841 raft_consensus.cc:1261] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Refusing update from remote peer ff45df6ea44a4d4c9d74d92c8f07b71d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:59.857884 10986 raft_consensus.cc:1261] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 FOLLOWER]: Refusing update from remote peer ff45df6ea44a4d4c9d74d92c8f07b71d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:59.858076 11232 consensus_queue.cc:1035] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:59.858218 11199 consensus_queue.cc:1035] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:59.858757 11196 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: TxnStatusTablet state changed. Reason: New leader ff45df6ea44a4d4c9d74d92c8f07b71d. Latest consensus state: current_term: 1 leader_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.858783 11200 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: TxnStatusTablet state changed. Reason: New leader ff45df6ea44a4d4c9d74d92c8f07b71d. Latest consensus state: current_term: 1 leader_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.858852 11196 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:25:59.858868 11200 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:25:59.859313 11199 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.859382 11199 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:25:59.859470 11207 mvcc.cc:205] Tried to move back new op lower bound from 7298518056375877632 to 7298518056310345728. Current Snapshot: MvccSnapshot[applied={T|T < 7298518056375877632}]
I20260619 10:25:59.859611 11196 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.859689 11196 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P b28dd69da5f045fa9f91af9ff6371dc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:25:59.860700 11232 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.860769 11232 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P ff45df6ea44a4d4c9d74d92c8f07b71d: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:25:59.861061 11200 tablet_replica.cc:439] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 } } peers { permanent_uuid: "26dc50861ec847098c4e34b1fc0d9c09" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 36851 } } peers { permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 } } }
I20260619 10:25:59.861160 11200 tablet_replica.cc:442] T d02f8e5068fb4dd68e87050d7d8f3d6a P 26dc50861ec847098c4e34b1fc0d9c09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:25:59.867890 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.862763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:48205 (local address 127.9.90.67:40737)
0619 10:25:59.862794 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.862796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.865965 (+ 3169us) server_negotiation.cc:407] Connection header received
0619 10:25:59.866010 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.866011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.866021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.866065 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:59.866167 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.866619 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.867330 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.867503 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.867553 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.867684 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:59.867715 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:59.867753 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.867809 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.867826 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:25:59.867913 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.862719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:40737 (local address 127.9.90.66:48205)
0619 10:25:59.865894 (+ 3175us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.865906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.865964 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.866072 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.866073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.866074 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:25:59.866119 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.866120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.866690 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.866691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.867259 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.867262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.867559 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.867560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.867610 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.867622 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:25:59.867760 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:25:59.867761 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.867801 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.867847 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3144,"thread_start_us":50,"threads_started":1}
I20260619 10:25:59.869923 11126 raft_consensus.cc:1261] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Refusing update from remote peer 26dc50861ec847098c4e34b1fc0d9c09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:59.870046 10841 raft_consensus.cc:1261] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Refusing update from remote peer 26dc50861ec847098c4e34b1fc0d9c09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:25:59.870124 11200 consensus_queue.cc:1035] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff45df6ea44a4d4c9d74d92c8f07b71d" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:59.870232 11202 consensus_queue.cc:1035] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b28dd69da5f045fa9f91af9ff6371dc7" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:25:59.881269 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.878875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:58671 (local address 127.9.90.66:36851)
0619 10:25:59.878979 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.878981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.879151 (+ 170us) server_negotiation.cc:407] Connection header received
0619 10:25:59.879190 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.879191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.879202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.879245 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:25:59.879357 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.879807 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.880538 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.880816 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.880873 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.881076 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:25:59.881112 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:25:59.881147 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.881199 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.881216 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 10:25:59.881337 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.878837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:36851 (local address 127.9.90.67:58671)
0619 10:25:59.879087 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 10:25:59.879096 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:25:59.879147 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:25:59.879245 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:25:59.879245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:25:59.879247 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:25:59.879305 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.879307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.879871 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.879872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.880461 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:25:59.880464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.880923 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.880924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:25:59.880996 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.881011 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:25:59.881151 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:25:59.881152 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:25:59.881198 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:25:59.881239 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
I20260619 10:25:59.887385 10688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:59.887471 10688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:25:59.888000 10688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edf526a026e3465ca1b563956a2a05a4: Sending DeleteTablet for 3 replicas of tablet f7d4d474356a4e9796a6c14e40c6c28a
I20260619 10:25:59.888218 10966 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d4d474356a4e9796a6c14e40c6c28a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:59 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:40906
I20260619 10:25:59.888265 10826 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d4d474356a4e9796a6c14e40c6c28a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:59 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:57210
I20260619 10:25:59.888368 11215 tablet_replica.cc:330] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: stopping tablet replica
I20260619 10:25:59.888373 11214 tablet_replica.cc:330] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: stopping tablet replica
I20260619 10:25:59.888420 11215 raft_consensus.cc:2229] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:59.888427 11214 raft_consensus.cc:2229] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:25:59.888458 11215 raft_consensus.cc:2258] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:59.888471 11214 raft_consensus.cc:2258] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:59.888592 11106 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d4d474356a4e9796a6c14e40c6c28a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:25:59 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:59296
I20260619 10:25:59.888685 11216 tablet_replica.cc:330] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: stopping tablet replica
I20260619 10:25:59.888697 11214 ts_tablet_manager.cc:1905] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:59.888731 11216 raft_consensus.cc:2229] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:25:59.888761 11216 raft_consensus.cc:2258] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:25:59.889043 11216 ts_tablet_manager.cc:1905] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:59.889492 11215 ts_tablet_manager.cc:1905] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:25:59.890324 11214 ts_tablet_manager.cc:1918] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:25:59.890379 11214 log.cc:1196] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/f7d4d474356a4e9796a6c14e40c6c28a
I20260619 10:25:59.890570 11214 ts_tablet_manager.cc:1939] T f7d4d474356a4e9796a6c14e40c6c28a P 26dc50861ec847098c4e34b1fc0d9c09: Deleting consensus metadata
I20260619 10:25:59.890647 11216 ts_tablet_manager.cc:1918] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:25:59.890684 11216 log.cc:1196] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/f7d4d474356a4e9796a6c14e40c6c28a
I20260619 10:25:59.890873 11216 ts_tablet_manager.cc:1939] T f7d4d474356a4e9796a6c14e40c6c28a P ff45df6ea44a4d4c9d74d92c8f07b71d: Deleting consensus metadata
I20260619 10:25:59.891070 11215 ts_tablet_manager.cc:1918] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:25:59.891125 11215 log.cc:1196] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/f7d4d474356a4e9796a6c14e40c6c28a
I20260619 10:25:59.891146 10675 catalog_manager.cc:4841] TS ff45df6ea44a4d4c9d74d92c8f07b71d (127.9.90.67:40737): tablet f7d4d474356a4e9796a6c14e40c6c28a (table test-table [id=91a88c3d6c03413381c6289cfe863978]) successfully deleted
I20260619 10:25:59.891146 10676 catalog_manager.cc:4841] TS 26dc50861ec847098c4e34b1fc0d9c09 (127.9.90.66:36851): tablet f7d4d474356a4e9796a6c14e40c6c28a (table test-table [id=91a88c3d6c03413381c6289cfe863978]) successfully deleted
I20260619 10:25:59.891316 11215 ts_tablet_manager.cc:1939] T f7d4d474356a4e9796a6c14e40c6c28a P b28dd69da5f045fa9f91af9ff6371dc7: Deleting consensus metadata
I20260619 10:25:59.891575 10675 catalog_manager.cc:4841] TS b28dd69da5f045fa9f91af9ff6371dc7 (127.9.90.65:36793): tablet f7d4d474356a4e9796a6c14e40c6c28a (table test-table [id=91a88c3d6c03413381c6289cfe863978]) successfully deleted
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:25:59.899699 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.891688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.9.90.65:36793)
0619 10:25:59.891730 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.891731 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.891973 (+ 242us) server_negotiation.cc:407] Connection header received
0619 10:25:59.891997 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.891998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.892008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.892062 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.892521 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.893244 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.893963 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.894257 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.894326 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.896801 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.896809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.896810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.896817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.898533 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.898934 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.898939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.898939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.899001 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.899159 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.899160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.899160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.899337 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.899477 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.899521 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.899598 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:25:59.899941 10886 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57212
I20260619 10:25:59.907136 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:25:59.900831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.9.90.126:34409)
0619 10:25:59.900876 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:25:59.900878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:25:59.900887 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:25:59.900915 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:25:59.900916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:25:59.900926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:25:59.900990 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:25:59.901437 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.902250 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.902871 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:25:59.903174 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:25:59.903239 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:25:59.904200 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:25:59.904201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:25:59.904201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:25:59.904208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:25:59.905596 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.905936 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.905937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.905937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.905995 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:25:59.906605 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:25:59.906605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:25:59.906606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:25:59.906750 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:25:59.906850 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 10:25:59.906914 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 10:25:59.907026 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 10:25:59.907307 10688 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46924
I20260619 10:25:59.907863 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 10761
I20260619 10:25:59.913587 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 10901
I20260619 10:25:59.918553 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 11041
I20260619 10:25:59.922562 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 10657
2026-06-19T10:25:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2247 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[11255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[11255](info): set up 2 sockets
Jun 19 10:25:59 dist-test-slave-hw7q krb5kdc[11255](info): commencing operation
krb5kdc: starting...
W20260619 10:26:00.869160 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:00 dist-test-slave-hw7q krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:00.972023 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:44325
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:01.051288 11268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:01.051611 11268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:01.051685 11268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:01.053326 11268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:01.053387 11268 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 10:26:01.053407 11268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:01.053434 11268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:01.053504 11268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:01.053535 11268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:01.055809 11268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:44325
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:01.056072 11268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:01.056311 11268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:01.061113 11273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:01.061182 11276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:01.061106 11274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:01.061458 11268 server_base.cc:1034] running on GCE node
I20260619 10:26:01.061689 11268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:01.061995 11268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:01.063148 11268 hybrid_clock.cc:648] HybridClock initialized: now 1781864761063129 us; error 36 us; skew 500 ppm
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:01.065332 11268 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:01.065732 11268 webserver.cc:460] Webserver started at http://127.9.90.126:34433/ using document root <none> and password file <none>
I20260619 10:26:01.065986 11268 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:01.066031 11268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:01.066148 11268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:01.067211 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "0d91ff395ffb4cb5bac39343ad403c79"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "a4fd4d372cc762903186061307786980"
server_key_iv: "3242392af0ff1aaf3d90d4b71d93e4cc"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.067525 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "0d91ff395ffb4cb5bac39343ad403c79"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "a4fd4d372cc762903186061307786980"
server_key_iv: "3242392af0ff1aaf3d90d4b71d93e4cc"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.068768 11268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:01.069365 11283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.069510 11268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:01.069592 11268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "0d91ff395ffb4cb5bac39343ad403c79"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "a4fd4d372cc762903186061307786980"
server_key_iv: "3242392af0ff1aaf3d90d4b71d93e4cc"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.069643 11268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:01.085067 11268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:01.086944 11268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:01.087054 11268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:01.091259 11268 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:44325
I20260619 10:26:01.091305 11345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:44325 every 8 connection(s)
I20260619 10:26:01.091612 11268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:01.092226 11346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.094445 11346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: Bootstrap starting.
I20260619 10:26:01.094964 11346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.095202 11346 log.cc:824] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:01.095798 11346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: No bootstrap required, opened a new log
I20260619 10:26:01.096042 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 11268
I20260619 10:26:01.096124 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:01.096271 9577 external_mini_cluster.cc:1428] Setting key 8ed7671d06ed48ba1bac2c392d5243aa
I20260619 10:26:01.098052 11346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91ff395ffb4cb5bac39343ad403c79" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44325 } }
I20260619 10:26:01.098191 11346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.098222 11346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d91ff395ffb4cb5bac39343ad403c79, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.098387 11346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91ff395ffb4cb5bac39343ad403c79" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44325 } }
I20260619 10:26:01.098469 11346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:01.098506 11346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:01.098552 11346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.099099 11346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91ff395ffb4cb5bac39343ad403c79" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44325 } }
I20260619 10:26:01.099181 11346 leader_election.cc:302] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d91ff395ffb4cb5bac39343ad403c79; no voters:
I20260619 10:26:01.099327 11346 leader_election.cc:288] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:01.099366 11351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:01.099517 11351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [term 1 LEADER]: Becoming Leader. State: Replica: 0d91ff395ffb4cb5bac39343ad403c79, State: Running, Role: LEADER
I20260619 10:26:01.099575 11346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:01.100698 11351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91ff395ffb4cb5bac39343ad403c79" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44325 } }
I20260619 10:26:01.101280 11352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d91ff395ffb4cb5bac39343ad403c79. Latest consensus state: current_term: 1 leader_uuid: "0d91ff395ffb4cb5bac39343ad403c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91ff395ffb4cb5bac39343ad403c79" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44325 } } }
I20260619 10:26:01.101452 11353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d91ff395ffb4cb5bac39343ad403c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91ff395ffb4cb5bac39343ad403c79" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44325 } } }
I20260619 10:26:01.101696 11352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:01.101722 11353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79 [sys.catalog]: This master's current role is: LEADER
W20260619 10:26:01.101610 11365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:26:01.101773 11365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:26:01.102780 11368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:01.102954 11368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:01.104324 11368 catalog_manager.cc:1347] Generated new cluster ID: 5fefc24bf104497bb3376aca0d3cf6c9
I20260619 10:26:01.104377 11368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:01.107201 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.096715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.9.90.126:44325)
0619 10:26:01.096869 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.096877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.096901 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:26:01.098024 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.098030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.098074 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.098182 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:01.098888 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.099989 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.100735 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.101284 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.101705 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.103819 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:01.103823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:01.103824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:01.103835 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:01.105658 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.106095 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.106097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.106098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.106202 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.106404 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.106405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.106405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.106661 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:01.106800 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.106899 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.107019 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260619 10:26:01.110725 11368 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:01.111207 11368 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:01.114605 11368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: Generated new TSK 0
I20260619 10:26:01.114765 11368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:26:01.140693 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:44325
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:01.238557 11372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:01.238762 11372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:01.238802 11372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:01.239931 11372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:01.239971 11372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:01.239998 11372 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 10:26:01.240012 11372 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:26:01.240047 11372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:01.241472 11372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.90.126:44325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:01.241691 11372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:01.241951 11372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:01.246451 11380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:01.246903 11377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:01.246912 11378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:01.247057 11372 server_base.cc:1034] running on GCE node
I20260619 10:26:01.247280 11372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:01.247493 11372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:01.248639 11372 hybrid_clock.cc:648] HybridClock initialized: now 1781864761248623 us; error 34 us; skew 500 ppm
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:01.250980 11372 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:26:01.251415 11372 webserver.cc:460] Webserver started at http://127.9.90.65:45405/ using document root <none> and password file <none>
I20260619 10:26:01.251617 11372 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:01.251658 11372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:01.251780 11372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:01.252808 11372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "19c532bbc77d4744bb207834b6c0f476"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "f46df2ab471bc677f0d698e3735d878e"
server_key_iv: "dc3c82c6e70af793cfed26ae1730d27c"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.253125 11372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "19c532bbc77d4744bb207834b6c0f476"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "f46df2ab471bc677f0d698e3735d878e"
server_key_iv: "dc3c82c6e70af793cfed26ae1730d27c"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.254453 11372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:01.255162 11387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.255380 11372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:01.255463 11372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "19c532bbc77d4744bb207834b6c0f476"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "f46df2ab471bc677f0d698e3735d878e"
server_key_iv: "dc3c82c6e70af793cfed26ae1730d27c"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.255517 11372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:01.267596 11372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:01.269151 11372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:01.269249 11372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:01.269652 11372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:01.269701 11372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.269719 11372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:01.269732 11372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.275579 11372 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:45843
I20260619 10:26:01.275699 11502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:45843 every 8 connection(s)
I20260619 10:26:01.276006 11372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:01.279125 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.270080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:49471 (local address 127.9.90.126:44325)
0619 10:26:01.270139 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.270141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.270308 (+ 167us) server_negotiation.cc:407] Connection header received
0619 10:26:01.270746 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.270747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.270762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.270842 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:01.271628 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.272408 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.273871 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.274174 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.274262 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.276603 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:01.276604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:01.276605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:01.276611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:01.277826 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.278268 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.278269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.278270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.278343 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.278572 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.278572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.278573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.278674 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:01.278720 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.278990 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.279059 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:26:01.279338 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.270053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44325 (local address 127.9.90.65:49471)
0619 10:26:01.270188 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.270224 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.270664 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.270939 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.270943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.271267 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:01.271532 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.271538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.272490 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.272491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.273763 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.273766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.274267 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.274268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.274366 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.274860 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:01.274870 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:01.276520 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:01.277909 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.277915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.277917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.278183 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.278404 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.278405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.278405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.278519 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.278720 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:01.278724 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:01.278873 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.278974 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.279126 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260619 10:26:01.279804 11503 heartbeater.cc:347] Connected to a master server at 127.9.90.126:44325
I20260619 10:26:01.279879 11503 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:01.280045 11503 heartbeater.cc:511] Master 127.9.90.126:44325 requested a full tablet report, sending...
I20260619 10:26:01.280567 11299 ts_manager.cc:194] Registered new tserver with Master: 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843)
I20260619 10:26:01.281422 11299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:49471
I20260619 10:26:01.285650 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.280869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:41779 (local address 127.9.90.126:44325)
0619 10:26:01.280903 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.280905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.280998 (+ 93us) server_negotiation.cc:407] Connection header received
0619 10:26:01.281057 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.281058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.281064 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.281116 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:01.281434 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.282019 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.282668 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.282856 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.282902 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.283875 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:01.283876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:01.283876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:01.283882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:01.284890 (+ 1008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.285158 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.285159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.285159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.285203 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.285322 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.285323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.285323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.285404 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:01.285443 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.285538 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.285588 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:26:01.285704 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.280812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44325 (local address 127.9.90.65:41779)
0619 10:26:01.280950 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.280960 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.281008 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.281118 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.281118 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.281288 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:01.281353 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.281354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.282102 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.282103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.282609 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.282611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.282931 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.282932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.283001 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.283432 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:01.283437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:01.283778 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:01.284950 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.284951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.284951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.285095 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.285247 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.285248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.285248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.285290 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.285440 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:01.285441 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:01.285472 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.285523 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.285613 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260619 10:26:01.285825 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 11372
I20260619 10:26:01.285912 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:01.286063 9577 external_mini_cluster.cc:1428] Setting key de47d8816d31ec5ddafcb2c95977ada4
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:26:01.306005 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:44325
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:01.384080 11512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:01.384284 11512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:01.384317 11512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:01.385414 11512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:01.385459 11512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:01.385489 11512 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 10:26:01.385509 11512 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:26:01.385542 11512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:01.386997 11512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.90.126:44325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:01.387224 11512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:01.387447 11512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:01.392066 11517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:01.392077 11520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:01.392184 11512 server_base.cc:1034] running on GCE node
W20260619 10:26:01.392132 11518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:01.392431 11512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:01.392613 11512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:01.393741 11512 hybrid_clock.cc:648] HybridClock initialized: now 1781864761393727 us; error 27 us; skew 500 ppm
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:01.395951 11512 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:26:01.396330 11512 webserver.cc:460] Webserver started at http://127.9.90.66:41021/ using document root <none> and password file <none>
I20260619 10:26:01.396525 11512 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:01.396569 11512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:01.396687 11512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:01.397612 11512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "d53dc9893c6b448f9f34ea5bfbae72b6"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "24b8d26a324b0ac78f37224c552ce8b6"
server_key_iv: "50aeb2493a7e5004e69816fcf8812e11"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.397944 11512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "d53dc9893c6b448f9f34ea5bfbae72b6"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "24b8d26a324b0ac78f37224c552ce8b6"
server_key_iv: "50aeb2493a7e5004e69816fcf8812e11"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.399081 11512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:01.399747 11527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.399894 11512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:01.399969 11512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "d53dc9893c6b448f9f34ea5bfbae72b6"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "24b8d26a324b0ac78f37224c552ce8b6"
server_key_iv: "50aeb2493a7e5004e69816fcf8812e11"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.400035 11512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:01.421458 11512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:01.423228 11512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:01.423367 11512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:01.423830 11512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:01.423868 11512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.423898 11512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:01.423920 11512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:01.430389 11512 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:45563
I20260619 10:26:01.430456 11642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:45563 every 8 connection(s)
I20260619 10:26:01.430751 11512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:01.432432 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.424302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:52137 (local address 127.9.90.126:44325)
0619 10:26:01.424371 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.424373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.424569 (+ 196us) server_negotiation.cc:407] Connection header received
0619 10:26:01.425029 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.425030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.425043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.425109 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:01.425928 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.426421 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.427349 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.427640 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.427720 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.429736 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:01.429738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:01.429738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:01.429745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:01.431249 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.431587 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.431588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.431588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.431645 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.431853 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.431854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.431854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.431960 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:01.432012 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.432262 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.432335 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:26:01.432570 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.424297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44325 (local address 127.9.90.66:52137)
0619 10:26:01.424450 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.424488 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.424942 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.425191 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.425194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.425516 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:01.425816 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.425824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.426502 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.426503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.427284 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.427286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.427743 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.427744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.427871 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.428342 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:01.428350 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:01.429653 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:01.431314 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.431319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.431321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.431520 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.431696 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.431696 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.431697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.431800 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.432010 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:01.432015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:01.432143 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.432228 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.432376 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260619 10:26:01.432969 11643 heartbeater.cc:347] Connected to a master server at 127.9.90.126:44325
I20260619 10:26:01.433044 11643 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:01.433200 11643 heartbeater.cc:511] Master 127.9.90.126:44325 requested a full tablet report, sending...
I20260619 10:26:01.433535 11300 ts_manager.cc:194] Registered new tserver with Master: d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563)
I20260619 10:26:01.434031 11300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:52137
I20260619 10:26:01.438555 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.433865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:48937 (local address 127.9.90.126:44325)
0619 10:26:01.433903 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.433905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.433997 (+ 92us) server_negotiation.cc:407] Connection header received
0619 10:26:01.434056 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.434057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.434064 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.434113 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:01.434433 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.434872 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.435518 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.435711 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.435760 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.436484 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:01.436486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:01.436486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:01.436492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:01.437550 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.437857 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.437858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.437858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.437913 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.438078 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.438079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.438080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.438212 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:01.438273 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.438401 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.438464 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:01.438565 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.433815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44325 (local address 127.9.90.66:48937)
0619 10:26:01.433944 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.433954 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.434009 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.434116 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.434116 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.434298 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:01.434362 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.434363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.434957 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.434958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.435453 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.435455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.435769 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.435769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.435840 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.436137 (+ 297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:01.436140 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:01.436412 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:01.437623 (+ 1211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.437624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.437624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.437805 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.437965 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.437965 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.437966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.438030 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.438267 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:01.438268 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:01.438314 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.438373 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.438467 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260619 10:26:01.440135 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 11512
I20260619 10:26:01.440209 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:01.440379 9577 external_mini_cluster.cc:1428] Setting key 0e92f840186120eda51d08667f06c29c
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:26:01.461009 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:44325
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:01.538496 11652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:01.538751 11652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:01.538795 11652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:01.539953 11652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:01.539994 11652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:01.540022 11652 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 10:26:01.540037 11652 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:26:01.540060 11652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:01.541447 11652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.90.126:44325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:01.541674 11652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:01.541908 11652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:01.546752 11657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:01.546779 11660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:01.546763 11658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:01.546830 11652 server_base.cc:1034] running on GCE node
I20260619 10:26:01.547127 11652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:01.547345 11652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:01.548513 11652 hybrid_clock.cc:648] HybridClock initialized: now 1781864761548498 us; error 28 us; skew 500 ppm
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:01.550846 11652 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:26:01.551276 11652 webserver.cc:460] Webserver started at http://127.9.90.67:41273/ using document root <none> and password file <none>
I20260619 10:26:01.551473 11652 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:01.551520 11652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:01.551638 11652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:01.552620 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "07c668b2331d41dabd1f098a62ccb646"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "9a6a788e37cf2fca5810331d332e1b12"
server_key_iv: "4cabeaa723ecc816f008313d399a3c9d"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.552943 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "07c668b2331d41dabd1f098a62ccb646"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "9a6a788e37cf2fca5810331d332e1b12"
server_key_iv: "4cabeaa723ecc816f008313d399a3c9d"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.554214 11652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:01.554921 11667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.555069 11652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:01.555154 11652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "07c668b2331d41dabd1f098a62ccb646"
format_stamp: "Formatted at 2026-06-19 10:26:01 on dist-test-slave-hw7q"
server_key: "9a6a788e37cf2fca5810331d332e1b12"
server_key_iv: "4cabeaa723ecc816f008313d399a3c9d"
server_key_version: "encryptionkey@0"
I20260619 10:26:01.555212 11652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:01.577083 11652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:01.578981 11652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:01.579126 11652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:01.579618 11652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:01.579663 11652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.579685 11652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:01.579697 11652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.585667 11652 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:38687
I20260619 10:26:01.585829 11782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:38687 every 8 connection(s)
I20260619 10:26:01.586062 11652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:01.588868 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.580082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:48487 (local address 127.9.90.126:44325)
0619 10:26:01.580137 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.580138 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.580399 (+ 261us) server_negotiation.cc:407] Connection header received
0619 10:26:01.580924 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.580926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.580941 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.581005 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:01.581815 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.582568 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.583997 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.584243 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.584309 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.586591 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:01.586592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:01.586593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:01.586599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:01.587752 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.588110 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.588111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.588111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.588159 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.588354 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.588354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.588354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.588444 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:01.588487 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.588700 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.588783 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:26:01.588950 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.580097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44325 (local address 127.9.90.67:48487)
0619 10:26:01.580266 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.580312 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.580804 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.581095 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.581100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.581428 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:01.581695 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.581701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.582664 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.582665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.583923 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.583926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.584319 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.584320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.584446 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.584973 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:01.584983 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:01.586523 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:01.587840 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.587846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.587847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.588049 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.588197 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.588198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.588198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.588312 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.588491 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:01.588494 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:01.588596 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.588670 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.588790 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260619 10:26:01.589390 11783 heartbeater.cc:347] Connected to a master server at 127.9.90.126:44325
I20260619 10:26:01.589465 11783 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:01.589618 11783 heartbeater.cc:511] Master 127.9.90.126:44325 requested a full tablet report, sending...
I20260619 10:26:01.589972 11300 ts_manager.cc:194] Registered new tserver with Master: 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67:38687)
I20260619 10:26:01.590399 11300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:48487
I20260619 10:26:01.594913 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.590386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:51803 (local address 127.9.90.126:44325)
0619 10:26:01.590514 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.590516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.590523 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:01.590538 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.590539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.590547 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.590590 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:01.590881 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.591324 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.591962 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.592137 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.592186 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.592923 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:01.592924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:01.592924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:01.592931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:01.594000 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.594270 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.594271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.594271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.594339 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.594495 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.594495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.594496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.594585 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:01.594635 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.594761 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.594825 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 10:26:01.594928 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.590304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44325 (local address 127.9.90.67:51803)
0619 10:26:01.590438 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.590449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.590498 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.590603 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.590604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.590768 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:01.590820 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.590821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.591402 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.591403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.591899 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.591900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.592189 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.592190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.592259 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.592595 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:01.592599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:01.592854 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:01.594070 (+ 1216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.594071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.594071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.594216 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.594392 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.594393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.594393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.594447 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.594634 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:01.594635 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:01.594677 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.594736 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.594828 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260619 10:26:01.595453 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 11652
I20260619 10:26:01.595544 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:01.595749 9577 external_mini_cluster.cc:1428] Setting key b04052a41de505e0723a193719043138
I20260619 10:26:01.596554 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:01.616953 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.610882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.9.90.126:44325)
0619 10:26:01.610941 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.610943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.610975 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:26:01.610998 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.610998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.611006 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.611045 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:01.611536 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.612025 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.612744 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.612931 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.612972 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.615033 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:01.615034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:01.615034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:01.615040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:01.616125 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.616446 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.616447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.616447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.616489 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.616611 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.616612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.616612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.616696 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:01.616761 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.616832 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.616883 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:26:01.618382 11300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:01.619807 11300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:01.625721 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.621932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.9.90.66:45563)
0619 10:26:01.622090 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.622095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.622325 (+ 230us) server_negotiation.cc:407] Connection header received
0619 10:26:01.622433 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.622436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.622452 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.622519 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:01.622690 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.623382 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.624734 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.625420 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.625474 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.625520 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.625601 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.625634 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260619 10:26:01.625699 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.621729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38687 (local address 127.0.0.1:51998)
0619 10:26:01.621920 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.621956 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.622017 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.622335 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.622338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.622340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:01.622396 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.622398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.623647 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.623648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.624727 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.624729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.625337 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.625338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.625421 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.625434 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.625497 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.625548 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
I20260619 10:26:01.625733 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.621998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.9.90.67:38687)
0619 10:26:01.622177 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.622182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.622192 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:01.622227 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.622228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.622240 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.622305 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:01.622460 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.623563 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.624782 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.625258 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.625318 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.625378 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.625607 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.625646 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260619 10:26:01.626025 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.622109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.9.90.65:45843)
0619 10:26:01.622240 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.622244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.622254 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:01.622293 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.622296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.622310 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.622381 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:01.622570 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.623611 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.624541 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.625265 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.625331 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.625383 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.625920 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.625951 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260619 10:26:01.625705 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.621731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45563 (local address 127.0.0.1:46652)
0619 10:26:01.622257 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.622268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.622354 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.622577 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.622578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.622579 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:01.622614 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.622615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.623457 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.623458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.624654 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.624656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.625488 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.625489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.625541 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.625544 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.625588 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.625623 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":2,"thread_start_us":121,"threads_started":1}
I20260619 10:26:01.626251 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.621731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:45843 (local address 127.0.0.1:35980)
0619 10:26:01.622111 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.622119 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.622152 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.622442 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.622443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.622444 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:01.622493 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.622495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.623689 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.623690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.624458 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.624461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.625419 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.625419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.625478 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.625482 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.625524 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.625563 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":72,"threads_started":1}
I20260619 10:26:01.626459 11717 tablet_service.cc:1467] Processing CreateTablet for tablet 5506576b79d34a228b208078861a7563 (DEFAULT_TABLE table=test-table [id=80c0bd8f9b3d4f199b3c99d3b054334c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:01.626847 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5506576b79d34a228b208078861a7563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.626886 11437 tablet_service.cc:1467] Processing CreateTablet for tablet 5506576b79d34a228b208078861a7563 (DEFAULT_TABLE table=test-table [id=80c0bd8f9b3d4f199b3c99d3b054334c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:01.627190 11437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5506576b79d34a228b208078861a7563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.629174 11806 tablet_bootstrap.cc:492] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: Bootstrap starting.
I20260619 10:26:01.629835 11807 tablet_bootstrap.cc:492] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: Bootstrap starting.
I20260619 10:26:01.630167 11806 tablet_bootstrap.cc:654] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.630117 11577 tablet_service.cc:1467] Processing CreateTablet for tablet 5506576b79d34a228b208078861a7563 (DEFAULT_TABLE table=test-table [id=80c0bd8f9b3d4f199b3c99d3b054334c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:01.630383 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5506576b79d34a228b208078861a7563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.630412 11807 tablet_bootstrap.cc:654] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.630483 11806 log.cc:824] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:01.630699 11807 log.cc:824] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:01.631136 11806 tablet_bootstrap.cc:492] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: No bootstrap required, opened a new log
I20260619 10:26:01.631193 11806 ts_tablet_manager.cc:1397] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:01.632450 11806 raft_consensus.cc:348] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.632575 11806 raft_consensus.cc:374] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.632607 11806 raft_consensus.cc:729] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c668b2331d41dabd1f098a62ccb646, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.632741 11806 consensus_queue.cc:260] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.632966 11806 ts_tablet_manager.cc:1428] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:26:01.632992 11783 heartbeater.cc:503] Master 127.9.90.126:44325 was elected leader, sending a full tablet report...
I20260619 10:26:01.633229 11811 tablet_bootstrap.cc:492] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: Bootstrap starting.
I20260619 10:26:01.633626 11807 tablet_bootstrap.cc:492] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: No bootstrap required, opened a new log
I20260619 10:26:01.633728 11807 ts_tablet_manager.cc:1397] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 10:26:01.633836 11811 tablet_bootstrap.cc:654] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.634141 11811 log.cc:824] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:01.634809 11811 tablet_bootstrap.cc:492] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: No bootstrap required, opened a new log
I20260619 10:26:01.634873 11811 ts_tablet_manager.cc:1397] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:01.635644 11807 raft_consensus.cc:348] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.635809 11807 raft_consensus.cc:374] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.635852 11807 raft_consensus.cc:729] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c532bbc77d4744bb207834b6c0f476, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.635954 11807 consensus_queue.cc:260] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.636200 11807 ts_tablet_manager.cc:1428] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:26:01.636229 11503 heartbeater.cc:503] Master 127.9.90.126:44325 was elected leader, sending a full tablet report...
I20260619 10:26:01.636646 11811 raft_consensus.cc:348] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.636778 11811 raft_consensus.cc:374] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.636816 11811 raft_consensus.cc:729] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53dc9893c6b448f9f34ea5bfbae72b6, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.636919 11811 consensus_queue.cc:260] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.637148 11811 ts_tablet_manager.cc:1428] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:26:01.637260 11643 heartbeater.cc:503] Master 127.9.90.126:44325 was elected leader, sending a full tablet report...
W20260619 10:26:01.681315 11644 tablet.cc:2369] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:26:01.706574 11813 raft_consensus.cc:482] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:01.706732 11813 raft_consensus.cc:504] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.707149 11813 leader_election.cc:288] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563), 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67:38687)
I20260619 10:26:01.709903 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.707500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:52431 (local address 127.9.90.66:45563)
0619 10:26:01.707548 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.707550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.707558 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:01.707586 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.707587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.707597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.707661 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:01.707810 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.708278 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.709213 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.709577 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.709635 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.709646 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.709778 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.709838 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:26:01.709924 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.707324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45563 (local address 127.9.90.65:52431)
0619 10:26:01.707444 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.707455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.707502 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.707677 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.707678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.707680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:01.707750 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.707752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.708352 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.708353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.709123 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.709125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.709647 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.709648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.709720 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.709723 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.709765 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.709827 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":21}
I20260619 10:26:01.709925 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.707416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:47697 (local address 127.9.90.67:38687)
0619 10:26:01.707474 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.707476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.707578 (+ 102us) server_negotiation.cc:407] Connection header received
0619 10:26:01.707642 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.707643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.707655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.707700 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:01.707834 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.708302 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.709217 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.709596 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.709658 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.709668 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.709825 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.709862 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:26:01.709949 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.707324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38687 (local address 127.9.90.65:47697)
0619 10:26:01.707524 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.707532 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.707580 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.707702 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.707703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.707705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:01.707779 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.707780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.708373 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.708374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.709135 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.709138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.709672 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.709673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.709741 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.709744 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.709804 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.709846 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":24,"thread_start_us":113,"threads_started":1}
I20260619 10:26:01.710260 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5506576b79d34a228b208078861a7563" candidate_uuid: "19c532bbc77d4744bb207834b6c0f476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c668b2331d41dabd1f098a62ccb646" is_pre_election: true
I20260619 10:26:01.710397 11737 raft_consensus.cc:2454] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19c532bbc77d4744bb207834b6c0f476 in term 0.
I20260619 10:26:01.710479 11597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5506576b79d34a228b208078861a7563" candidate_uuid: "19c532bbc77d4744bb207834b6c0f476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" is_pre_election: true
I20260619 10:26:01.710625 11597 raft_consensus.cc:2454] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19c532bbc77d4744bb207834b6c0f476 in term 0.
I20260619 10:26:01.710608 11391 leader_election.cc:302] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c668b2331d41dabd1f098a62ccb646, 19c532bbc77d4744bb207834b6c0f476; no voters:
I20260619 10:26:01.710755 11813 raft_consensus.cc:2781] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:01.710826 11813 raft_consensus.cc:482] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:01.710850 11813 raft_consensus.cc:3037] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.711516 11813 raft_consensus.cc:504] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.711695 11813 leader_election.cc:288] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [CANDIDATE]: Term 1 election: Requested vote from peers d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563), 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67:38687)
I20260619 10:26:01.711781 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5506576b79d34a228b208078861a7563" candidate_uuid: "19c532bbc77d4744bb207834b6c0f476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c668b2331d41dabd1f098a62ccb646"
I20260619 10:26:01.711786 11597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5506576b79d34a228b208078861a7563" candidate_uuid: "19c532bbc77d4744bb207834b6c0f476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6"
I20260619 10:26:01.711844 11737 raft_consensus.cc:3037] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.711846 11597 raft_consensus.cc:3037] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.712416 11597 raft_consensus.cc:2454] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19c532bbc77d4744bb207834b6c0f476 in term 1.
I20260619 10:26:01.712515 11737 raft_consensus.cc:2454] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19c532bbc77d4744bb207834b6c0f476 in term 1.
I20260619 10:26:01.712899 11389 leader_election.cc:302] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19c532bbc77d4744bb207834b6c0f476, d53dc9893c6b448f9f34ea5bfbae72b6; no voters:
I20260619 10:26:01.713011 11813 raft_consensus.cc:2781] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:01.713163 11813 raft_consensus.cc:686] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 1 LEADER]: Becoming Leader. State: Replica: 19c532bbc77d4744bb207834b6c0f476, State: Running, Role: LEADER
I20260619 10:26:01.713248 11813 consensus_queue.cc:237] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.714035 11300 catalog_manager.cc:5495] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65). New cstate: current_term: 1 leader_uuid: "19c532bbc77d4744bb207834b6c0f476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } health_report { overall_health: UNKNOWN } } }
Jun 19 10:26:01 dist-test-slave-hw7q krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:01.765489 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.758654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58050 (local address 127.9.90.126:44325)
0619 10:26:01.758703 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.758705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.758839 (+ 134us) server_negotiation.cc:407] Connection header received
0619 10:26:01.758865 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.758866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.758876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.758924 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:01.759394 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.759865 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.760510 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.760685 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.760736 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.763074 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:01.763075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:01.763075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:01.763081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:01.764485 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.764851 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.764851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.764852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.764894 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:01.765036 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:01.765036 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:01.765036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:01.765139 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:01.765171 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.765329 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.765397 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:26:01.765724 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.758575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44325 (local address 127.0.0.1:58050)
0619 10:26:01.758718 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.758729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.758823 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.758933 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.758934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.759271 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:01.759318 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.759321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.759933 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.759934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.760444 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.760446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.760760 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.760761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.760811 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.761366 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:01.761371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:01.762989 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:01.764559 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.764566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.764569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.764785 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.764944 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:01.764944 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:01.764945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:01.764987 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:01.765202 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:01.765208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:01.765243 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.765561 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.765645 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260619 10:26:01.770913 11300 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:58050:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 10:26:01.771116 11300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:01.772576 11437 tablet_service.cc:1467] Processing CreateTablet for tablet c78ea7c23e7d4991a1bf5aabb8ec19cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa83698208b44a079b8d13764bf195a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:26:01.772586 11717 tablet_service.cc:1467] Processing CreateTablet for tablet c78ea7c23e7d4991a1bf5aabb8ec19cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa83698208b44a079b8d13764bf195a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:26:01.772742 11437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78ea7c23e7d4991a1bf5aabb8ec19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.772742 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78ea7c23e7d4991a1bf5aabb8ec19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.773976 11577 tablet_service.cc:1467] Processing CreateTablet for tablet c78ea7c23e7d4991a1bf5aabb8ec19cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa83698208b44a079b8d13764bf195a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:26:01.774127 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78ea7c23e7d4991a1bf5aabb8ec19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.774282 11806 tablet_bootstrap.cc:492] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: Bootstrap starting.
I20260619 10:26:01.774277 11807 tablet_bootstrap.cc:492] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: Bootstrap starting.
I20260619 10:26:01.774638 11807 tablet_bootstrap.cc:654] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.774834 11806 tablet_bootstrap.cc:654] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.775177 11807 tablet_bootstrap.cc:492] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: No bootstrap required, opened a new log
I20260619 10:26:01.775216 11807 ts_tablet_manager.cc:1397] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:01.775362 11807 raft_consensus.cc:348] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.775422 11807 raft_consensus.cc:374] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.775418 11806 tablet_bootstrap.cc:492] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: No bootstrap required, opened a new log
I20260619 10:26:01.775434 11807 raft_consensus.cc:729] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c532bbc77d4744bb207834b6c0f476, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.775457 11806 ts_tablet_manager.cc:1397] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:01.775470 11807 consensus_queue.cc:260] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.775550 11807 ts_tablet_manager.cc:1428] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.775542 11811 tablet_bootstrap.cc:492] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: Bootstrap starting.
I20260619 10:26:01.775586 11806 raft_consensus.cc:348] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.775646 11806 raft_consensus.cc:374] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.775604 11813 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.775663 11806 raft_consensus.cc:729] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c668b2331d41dabd1f098a62ccb646, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.775678 11813 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:01.775702 11806 consensus_queue.cc:260] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.775777 11806 ts_tablet_manager.cc:1428] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.775885 11810 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.775959 11810 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:01.776201 11811 tablet_bootstrap.cc:654] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.776787 11811 tablet_bootstrap.cc:492] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: No bootstrap required, opened a new log
W20260619 10:26:01.776796 11504 tablet.cc:2369] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:26:01.776845 11811 ts_tablet_manager.cc:1397] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:01.776965 11811 raft_consensus.cc:348] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.777012 11811 raft_consensus.cc:374] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.777037 11811 raft_consensus.cc:729] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53dc9893c6b448f9f34ea5bfbae72b6, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.777077 11811 consensus_queue.cc:260] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.777155 11811 ts_tablet_manager.cc:1428] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.777215 11814 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.777292 11814 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:01.779067 11810 raft_consensus.cc:482] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:01.779134 11810 raft_consensus.cc:504] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.779435 11810 leader_election.cc:288] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843), d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563)
I20260619 10:26:01.782671 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.779805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:57465 (local address 127.9.90.65:45843)
0619 10:26:01.779845 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.779847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.779855 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:01.779875 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.779876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.779883 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.779924 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:01.780046 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.780766 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.782000 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.782365 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.782424 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.782434 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.782579 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.782612 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:01.782763 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.779580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45563 (local address 127.9.90.67:48895)
0619 10:26:01.779621 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.779632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.779675 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.779840 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.779841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.779842 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:01.779901 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.779902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.780618 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.780619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.781728 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.781730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.782522 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.782523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.782603 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.782606 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.782651 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.782692 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 10:26:01.782833 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.779613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:48895 (local address 127.9.90.66:45563)
0619 10:26:01.779651 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.779652 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.779661 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:26:01.779739 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.779739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.779748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.779795 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:01.779966 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.780545 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.781869 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.782413 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.782480 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.782489 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.782722 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.782772 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:26:01.782693 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.779609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:45843 (local address 127.9.90.67:57465)
0619 10:26:01.779767 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.779777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.779828 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.779926 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.779926 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.779928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:01.779980 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.779981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.780843 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.780844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.781933 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.781935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.782446 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.782447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.782522 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.782526 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.782568 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.782609 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260619 10:26:01.783115 11597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78ea7c23e7d4991a1bf5aabb8ec19cb" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" is_pre_election: true
I20260619 10:26:01.783197 11597 raft_consensus.cc:2454] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 0.
I20260619 10:26:01.783166 11457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78ea7c23e7d4991a1bf5aabb8ec19cb" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c532bbc77d4744bb207834b6c0f476" is_pre_election: true
I20260619 10:26:01.783264 11457 raft_consensus.cc:2454] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 0.
I20260619 10:26:01.783396 11669 leader_election.cc:302] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c668b2331d41dabd1f098a62ccb646, d53dc9893c6b448f9f34ea5bfbae72b6; no voters:
I20260619 10:26:01.783576 11810 raft_consensus.cc:2781] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:01.783609 11810 raft_consensus.cc:482] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:01.783627 11810 raft_consensus.cc:3037] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.784192 11810 raft_consensus.cc:504] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.784312 11810 leader_election.cc:288] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 election: Requested vote from peers 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843), d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563)
I20260619 10:26:01.784435 11597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78ea7c23e7d4991a1bf5aabb8ec19cb" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6"
I20260619 10:26:01.784435 11457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78ea7c23e7d4991a1bf5aabb8ec19cb" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c532bbc77d4744bb207834b6c0f476"
I20260619 10:26:01.784490 11457 raft_consensus.cc:3037] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.784490 11597 raft_consensus.cc:3037] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.785135 11597 raft_consensus.cc:2454] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 1.
I20260619 10:26:01.785157 11457 raft_consensus.cc:2454] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 1.
I20260619 10:26:01.785302 11669 leader_election.cc:302] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c668b2331d41dabd1f098a62ccb646, d53dc9893c6b448f9f34ea5bfbae72b6; no voters:
I20260619 10:26:01.785431 11810 raft_consensus.cc:2781] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:01.785591 11810 raft_consensus.cc:686] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [term 1 LEADER]: Becoming Leader. State: Replica: 07c668b2331d41dabd1f098a62ccb646, State: Running, Role: LEADER
I20260619 10:26:01.785670 11810 consensus_queue.cc:237] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.786106 11828 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: TxnStatusTablet state changed. Reason: New leader 07c668b2331d41dabd1f098a62ccb646. Latest consensus state: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.786230 11828 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:26:01.786391 11300 catalog_manager.cc:5495] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67). New cstate: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } health_report { overall_health: HEALTHY } } }
I20260619 10:26:01.786443 11830 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 10:26:01.786536 11830 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 10:26:01.792171 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.789004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38687 (local address 127.0.0.1:52006)
0619 10:26:01.789056 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.789066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.789111 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.789300 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.789301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.789302 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:01.789346 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.789348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.790241 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.790242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.791142 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.791145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.791634 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.791635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.791696 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.791712 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.792004 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.792012 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.792050 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.792093 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 10:26:01.792407 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.789134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.9.90.67:38687)
0619 10:26:01.789173 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.789175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.789182 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:01.789206 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.789207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.789223 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.789292 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:01.789397 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.790116 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.791246 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.791539 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.791597 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.791810 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.791916 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.791980 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.792326 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.792347 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:26:01.793972 11597 raft_consensus.cc:1261] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Refusing update from remote peer 07c668b2331d41dabd1f098a62ccb646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:01.793980 11457 raft_consensus.cc:1261] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Refusing update from remote peer 07c668b2331d41dabd1f098a62ccb646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:01.794226 11828 consensus_queue.cc:1035] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:01.794373 11828 consensus_queue.cc:1035] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:01.795476 11813 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: TxnStatusTablet state changed. Reason: New leader 07c668b2331d41dabd1f098a62ccb646. Latest consensus state: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.795559 11813 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:01.795624 11814 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: TxnStatusTablet state changed. Reason: New leader 07c668b2331d41dabd1f098a62ccb646. Latest consensus state: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.795686 11814 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:01.796651 11810 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.796727 11810 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:26:01.797537 11814 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.797609 11814 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P d53dc9893c6b448f9f34ea5bfbae72b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:01.798110 11835 mvcc.cc:205] Tried to move back new op lower bound from 7298518064305836032 to 7298518064274849792. Current Snapshot: MvccSnapshot[applied={T|T < 7298518064305836032 or (T in {7298518064305836032})}]
I20260619 10:26:01.801466 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.798211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.9.90.65:45843)
0619 10:26:01.798250 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.798252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.798260 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:01.798327 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.798328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.798345 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.798467 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:01.798618 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.799344 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.800496 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.800793 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.800850 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.801037 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.801154 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.801213 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.801376 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.801397 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:01.802306 11828 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.802390 11828 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 07c668b2331d41dabd1f098a62ccb646: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:26:01.803050 11813 tablet_replica.cc:439] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } } }
I20260619 10:26:01.803135 11813 tablet_replica.cc:442] T c78ea7c23e7d4991a1bf5aabb8ec19cb P 19c532bbc77d4744bb207834b6c0f476: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:01.807368 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.804572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:51603 (local address 127.9.90.67:38687)
0619 10:26:01.804611 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.804612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.804620 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:01.804657 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.804658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.804666 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.804710 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:01.804833 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.805543 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.806621 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.806917 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.806983 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.807149 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.807181 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.807227 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.807278 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.807301 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:01.807399 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.804403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38687 (local address 127.9.90.65:51603)
0619 10:26:01.804545 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.804556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.804607 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.804713 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.804714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.804715 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:01.804777 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.804778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.805615 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.805615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.806544 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.806546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.806987 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.806988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.807049 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.807063 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.807232 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.807234 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.807275 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.807317 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260619 10:26:01.809617 11597 raft_consensus.cc:1261] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Refusing update from remote peer 19c532bbc77d4744bb207834b6c0f476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:01.809624 11737 raft_consensus.cc:1261] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Refusing update from remote peer 19c532bbc77d4744bb207834b6c0f476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:01.809860 11813 consensus_queue.cc:1035] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:01.810024 11816 consensus_queue.cc:1035] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:01.811118 11834 mvcc.cc:205] Tried to move back new op lower bound from 7298518064370561024 to 7298518063977988096. Current Snapshot: MvccSnapshot[applied={T|T < 7298518064370561024}]
I20260619 10:26:01.819064 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.816007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:45843 (local address 127.9.90.67:47293)
0619 10:26:01.816063 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.816073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.816112 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.816252 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.816253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.816254 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:01.816320 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.816322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.817185 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.817186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.818130 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.818132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.818665 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.818666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.818728 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.818742 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.818914 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.818916 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.818950 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.818988 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 10:26:01.819384 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.816104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:47293 (local address 127.9.90.65:45843)
0619 10:26:01.816135 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.816137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.816144 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:01.816163 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.816164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.816174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.816213 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:01.816385 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.817097 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.818215 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.818529 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.818641 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.818825 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.818857 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.818895 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.819294 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.819321 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:26:01.827606 11300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:01.827692 11300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:01.828310 11300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: Sending DeleteTablet for 3 replicas of tablet 5506576b79d34a228b208078861a7563
I20260619 10:26:01.828565 11577 tablet_service.cc:1514] Processing DeleteTablet for tablet 5506576b79d34a228b208078861a7563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:01 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46652
I20260619 10:26:01.828596 11437 tablet_service.cc:1514] Processing DeleteTablet for tablet 5506576b79d34a228b208078861a7563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:01 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260619 10:26:01.828926 11717 tablet_service.cc:1514] Processing DeleteTablet for tablet 5506576b79d34a228b208078861a7563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:01 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:51998
I20260619 10:26:01.829087 11300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:01.829208 11300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:01.830314 11437 tablet_service.cc:1467] Processing CreateTablet for tablet 24edd738f6b04fb68e24ffa04b74f300 (DEFAULT_TABLE table=test-table [id=d5a2547449524ce29e0066c134663179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:01.830329 11577 tablet_service.cc:1467] Processing CreateTablet for tablet 24edd738f6b04fb68e24ffa04b74f300 (DEFAULT_TABLE table=test-table [id=d5a2547449524ce29e0066c134663179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:01.830423 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24edd738f6b04fb68e24ffa04b74f300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.830423 11437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24edd738f6b04fb68e24ffa04b74f300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.831660 11807 tablet_bootstrap.cc:492] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: Bootstrap starting.
I20260619 10:26:01.831667 11811 tablet_bootstrap.cc:492] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: Bootstrap starting.
I20260619 10:26:01.832003 11811 tablet_bootstrap.cc:654] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.832054 11807 tablet_bootstrap.cc:654] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.832520 11717 tablet_service.cc:1467] Processing CreateTablet for tablet 24edd738f6b04fb68e24ffa04b74f300 (DEFAULT_TABLE table=test-table [id=d5a2547449524ce29e0066c134663179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:01.832614 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24edd738f6b04fb68e24ffa04b74f300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:01.832629 11811 tablet_bootstrap.cc:492] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: No bootstrap required, opened a new log
I20260619 10:26:01.832660 11811 ts_tablet_manager.cc:1397] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:01.832792 11811 raft_consensus.cc:348] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.832846 11811 raft_consensus.cc:374] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.832862 11811 raft_consensus.cc:729] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53dc9893c6b448f9f34ea5bfbae72b6, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.832897 11811 consensus_queue.cc:260] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.832978 11811 ts_tablet_manager.cc:1428] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.833835 11806 tablet_bootstrap.cc:492] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: Bootstrap starting.
I20260619 10:26:01.834209 11806 tablet_bootstrap.cc:654] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:01.834743 11806 tablet_bootstrap.cc:492] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: No bootstrap required, opened a new log
I20260619 10:26:01.834787 11806 ts_tablet_manager.cc:1397] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:01.834895 11806 raft_consensus.cc:348] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.834940 11806 raft_consensus.cc:374] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.834951 11806 raft_consensus.cc:729] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c668b2331d41dabd1f098a62ccb646, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.834983 11806 consensus_queue.cc:260] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.835057 11806 ts_tablet_manager.cc:1428] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260619 10:26:01.836536 11784 tablet.cc:2369] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:26:01.837982 11853 tablet_replica.cc:330] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: stopping tablet replica
I20260619 10:26:01.838088 11853 raft_consensus.cc:2229] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:01.838124 11807 tablet_bootstrap.cc:492] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: No bootstrap required, opened a new log
I20260619 10:26:01.838155 11853 raft_consensus.cc:2258] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:01.838160 11807 ts_tablet_manager.cc:1397] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260619 10:26:01.838308 11852 tablet_replica.cc:330] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: stopping tablet replica
I20260619 10:26:01.838294 11807 raft_consensus.cc:348] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.838353 11807 raft_consensus.cc:374] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:01.838371 11807 raft_consensus.cc:729] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c532bbc77d4744bb207834b6c0f476, State: Initialized, Role: FOLLOWER
I20260619 10:26:01.838388 11852 raft_consensus.cc:2229] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:01.838404 11807 consensus_queue.cc:260] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.838464 11852 raft_consensus.cc:2258] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:01.838470 11807 ts_tablet_manager.cc:1428] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:01.839345 11852 ts_tablet_manager.cc:1905] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:01.840507 11854 tablet_replica.cc:330] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: stopping tablet replica
I20260619 10:26:01.840600 11854 raft_consensus.cc:2229] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:01.840651 11854 raft_consensus.cc:2258] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:01.840878 11852 ts_tablet_manager.cc:1918] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:01.840947 11852 log.cc:1196] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/5506576b79d34a228b208078861a7563
I20260619 10:26:01.841020 11854 ts_tablet_manager.cc:1905] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:01.841187 11852 ts_tablet_manager.cc:1939] T 5506576b79d34a228b208078861a7563 P 19c532bbc77d4744bb207834b6c0f476: Deleting consensus metadata
I20260619 10:26:01.841567 11286 catalog_manager.cc:4841] TS 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843): tablet 5506576b79d34a228b208078861a7563 (table test-table [id=80c0bd8f9b3d4f199b3c99d3b054334c]) successfully deleted
I20260619 10:26:01.841665 11853 ts_tablet_manager.cc:1905] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:01.842394 11854 ts_tablet_manager.cc:1918] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:01.842453 11854 log.cc:1196] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/5506576b79d34a228b208078861a7563
I20260619 10:26:01.842649 11854 ts_tablet_manager.cc:1939] T 5506576b79d34a228b208078861a7563 P 07c668b2331d41dabd1f098a62ccb646: Deleting consensus metadata
I20260619 10:26:01.843021 11287 catalog_manager.cc:4841] TS 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67:38687): tablet 5506576b79d34a228b208078861a7563 (table test-table [id=80c0bd8f9b3d4f199b3c99d3b054334c]) successfully deleted
I20260619 10:26:01.843201 11853 ts_tablet_manager.cc:1918] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:01.843271 11853 log.cc:1196] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/5506576b79d34a228b208078861a7563
I20260619 10:26:01.843497 11853 ts_tablet_manager.cc:1939] T 5506576b79d34a228b208078861a7563 P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting consensus metadata
I20260619 10:26:01.843870 11285 catalog_manager.cc:4841] TS d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563): tablet 5506576b79d34a228b208078861a7563 (table test-table [id=80c0bd8f9b3d4f199b3c99d3b054334c]) successfully deleted
I20260619 10:26:01.871465 11850 raft_consensus.cc:482] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:01.871567 11850 raft_consensus.cc:504] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.871723 11850 leader_election.cc:288] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843), d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563)
I20260619 10:26:01.871937 11457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24edd738f6b04fb68e24ffa04b74f300" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c532bbc77d4744bb207834b6c0f476" is_pre_election: true
I20260619 10:26:01.872026 11457 raft_consensus.cc:2454] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 0.
I20260619 10:26:01.872179 11670 leader_election.cc:302] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c668b2331d41dabd1f098a62ccb646, 19c532bbc77d4744bb207834b6c0f476; no voters:
I20260619 10:26:01.872179 11596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24edd738f6b04fb68e24ffa04b74f300" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" is_pre_election: true
I20260619 10:26:01.872267 11596 raft_consensus.cc:2454] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 0.
I20260619 10:26:01.872290 11850 raft_consensus.cc:2781] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:01.872315 11850 raft_consensus.cc:482] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:01.872332 11850 raft_consensus.cc:3037] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.873061 11850 raft_consensus.cc:504] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.873178 11850 leader_election.cc:288] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 election: Requested vote from peers 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843), d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563)
I20260619 10:26:01.873373 11596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24edd738f6b04fb68e24ffa04b74f300" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6"
I20260619 10:26:01.873394 11457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24edd738f6b04fb68e24ffa04b74f300" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c532bbc77d4744bb207834b6c0f476"
I20260619 10:26:01.873441 11596 raft_consensus.cc:3037] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.873447 11457 raft_consensus.cc:3037] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:01.874354 11596 raft_consensus.cc:2454] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 1.
I20260619 10:26:01.874496 11457 raft_consensus.cc:2454] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 1.
I20260619 10:26:01.874651 11670 leader_election.cc:302] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c668b2331d41dabd1f098a62ccb646, 19c532bbc77d4744bb207834b6c0f476; no voters:
I20260619 10:26:01.874761 11850 raft_consensus.cc:2781] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:01.874809 11850 raft_consensus.cc:686] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 1 LEADER]: Becoming Leader. State: Replica: 07c668b2331d41dabd1f098a62ccb646, State: Running, Role: LEADER
I20260619 10:26:01.874857 11850 consensus_queue.cc:237] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:01.875383 11300 catalog_manager.cc:5495] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67). New cstate: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } health_report { overall_health: HEALTHY } } }
I20260619 10:26:01.883831 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.881858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.9.90.67:38687)
0619 10:26:01.881898 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.881900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.881906 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:26:01.881925 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.881926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.881935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.881980 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:01.882098 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.882561 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.883235 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.883433 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.883486 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.883603 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.883636 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.883678 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.883740 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.883758 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:26:01.887535 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.885618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38687 (local address 127.9.90.67:51565)
0619 10:26:01.885658 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:01.885668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:01.885700 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:01.885852 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:01.885852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:01.885853 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:01.885886 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.885887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.886415 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.886416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.886929 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:01.886930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.887226 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.887226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:01.887272 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.887276 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.887395 (+ 119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.887396 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:26:01.887441 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:26:01.887474 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 10:26:01.887535 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:01.885656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:51565 (local address 127.9.90.67:38687)
0619 10:26:01.885685 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:01.885686 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:01.885693 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:01.885758 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:01.885758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:01.885766 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:01.885848 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:01.885927 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.886357 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.886991 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:01.887175 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:01.887224 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:01.887329 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:01.887356 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:01.887396 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:01.887445 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 10:26:01.887458 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:26:01.889520 11457 raft_consensus.cc:1261] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Refusing update from remote peer 07c668b2331d41dabd1f098a62ccb646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:01.889520 11596 raft_consensus.cc:1261] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Refusing update from remote peer 07c668b2331d41dabd1f098a62ccb646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:01.889679 11850 consensus_queue.cc:1035] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:01.889807 11810 consensus_queue.cc:1035] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 10:26:01.895114 11833 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 10:26:03.414579 11300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:03.414676 11300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:03.415411 11300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: Sending DeleteTablet for 3 replicas of tablet 24edd738f6b04fb68e24ffa04b74f300
I20260619 10:26:03.415633 11717 tablet_service.cc:1514] Processing DeleteTablet for tablet 24edd738f6b04fb68e24ffa04b74f300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:03 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:51998
I20260619 10:26:03.415630 11437 tablet_service.cc:1514] Processing DeleteTablet for tablet 24edd738f6b04fb68e24ffa04b74f300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:03 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260619 10:26:03.415683 11577 tablet_service.cc:1514] Processing DeleteTablet for tablet 24edd738f6b04fb68e24ffa04b74f300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:03 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46652
I20260619 10:26:03.415841 11887 tablet_replica.cc:330] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: stopping tablet replica
I20260619 10:26:03.415897 11887 raft_consensus.cc:2229] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:03.415933 11887 raft_consensus.cc:2258] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:03.416133 11887 ts_tablet_manager.cc:1905] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:03.417366 11887 ts_tablet_manager.cc:1918] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:26:03.417418 11887 log.cc:1196] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/24edd738f6b04fb68e24ffa04b74f300
I20260619 10:26:03.417600 11887 ts_tablet_manager.cc:1939] T 24edd738f6b04fb68e24ffa04b74f300 P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting consensus metadata
I20260619 10:26:03.417881 11285 catalog_manager.cc:4841] TS d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563): tablet 24edd738f6b04fb68e24ffa04b74f300 (table test-table [id=d5a2547449524ce29e0066c134663179]) successfully deleted
I20260619 10:26:03.417898 11886 tablet_replica.cc:330] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: stopping tablet replica
I20260619 10:26:03.417953 11886 raft_consensus.cc:2229] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:03.418030 11886 raft_consensus.cc:2258] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:03.418125 11885 tablet_replica.cc:330] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: stopping tablet replica
I20260619 10:26:03.418190 11885 raft_consensus.cc:2229] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:03.418226 11885 raft_consensus.cc:2258] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:03.418280 11886 ts_tablet_manager.cc:1905] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:03.418464 11885 ts_tablet_manager.cc:1905] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:03.419567 11885 ts_tablet_manager.cc:1918] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:26:03.419646 11885 log.cc:1196] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/24edd738f6b04fb68e24ffa04b74f300
I20260619 10:26:03.419847 11886 ts_tablet_manager.cc:1918] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:26:03.419862 11885 ts_tablet_manager.cc:1939] T 24edd738f6b04fb68e24ffa04b74f300 P 19c532bbc77d4744bb207834b6c0f476: Deleting consensus metadata
I20260619 10:26:03.419910 11886 log.cc:1196] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/24edd738f6b04fb68e24ffa04b74f300
I20260619 10:26:03.420126 11886 ts_tablet_manager.cc:1939] T 24edd738f6b04fb68e24ffa04b74f300 P 07c668b2331d41dabd1f098a62ccb646: Deleting consensus metadata
I20260619 10:26:03.420415 11286 catalog_manager.cc:4841] TS 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843): tablet 24edd738f6b04fb68e24ffa04b74f300 (table test-table [id=d5a2547449524ce29e0066c134663179]) successfully deleted
I20260619 10:26:03.420576 11287 catalog_manager.cc:4841] TS 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67:38687): tablet 24edd738f6b04fb68e24ffa04b74f300 (table test-table [id=d5a2547449524ce29e0066c134663179]) successfully deleted
Jun 19 10:26:03 dist-test-slave-hw7q krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864763, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:26:03 dist-test-slave-hw7q krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864763, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:03.437559 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.431120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.9.90.126:44325)
0619 10:26:03.431240 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:03.431242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:03.431307 (+ 65us) server_negotiation.cc:407] Connection header received
0619 10:26:03.431378 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:03.431379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:03.431390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:03.431448 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:03.431955 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.432488 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.433173 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.433377 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.433429 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.435499 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:03.435500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:03.435500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:03.435508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:03.436600 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:03.436891 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:03.436892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:03.436892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:03.436944 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:03.437095 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:03.437096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:03.437097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:03.437199 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:03.437289 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:03.437389 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 10:26:03.437466 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260619 10:26:03.438956 11300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:03.439087 11300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:03.440303 11437 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbe18bc12f444b6ad537af41f3beead (DEFAULT_TABLE table=test-table [id=4e0258f62f4745e697e91e795fd01883]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:03.440307 11717 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbe18bc12f444b6ad537af41f3beead (DEFAULT_TABLE table=test-table [id=4e0258f62f4745e697e91e795fd01883]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:03.440306 11577 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbe18bc12f444b6ad537af41f3beead (DEFAULT_TABLE table=test-table [id=4e0258f62f4745e697e91e795fd01883]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:03.440426 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbe18bc12f444b6ad537af41f3beead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:03.440428 11437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbe18bc12f444b6ad537af41f3beead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:03.440446 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbe18bc12f444b6ad537af41f3beead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:03.441449 11896 tablet_bootstrap.cc:492] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: Bootstrap starting.
I20260619 10:26:03.441664 11897 tablet_bootstrap.cc:492] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: Bootstrap starting.
I20260619 10:26:03.441974 11898 tablet_bootstrap.cc:492] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: Bootstrap starting.
I20260619 10:26:03.442034 11896 tablet_bootstrap.cc:654] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:03.442034 11897 tablet_bootstrap.cc:654] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:03.442426 11898 tablet_bootstrap.cc:654] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:03.442631 11897 tablet_bootstrap.cc:492] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: No bootstrap required, opened a new log
I20260619 10:26:03.442667 11896 tablet_bootstrap.cc:492] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: No bootstrap required, opened a new log
I20260619 10:26:03.442677 11897 ts_tablet_manager.cc:1397] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:03.442708 11896 ts_tablet_manager.cc:1397] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:03.442800 11896 raft_consensus.cc:348] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.442848 11896 raft_consensus.cc:374] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:03.442806 11897 raft_consensus.cc:348] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.442865 11896 raft_consensus.cc:729] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53dc9893c6b448f9f34ea5bfbae72b6, State: Initialized, Role: FOLLOWER
I20260619 10:26:03.442876 11897 raft_consensus.cc:374] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:03.442898 11897 raft_consensus.cc:729] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c532bbc77d4744bb207834b6c0f476, State: Initialized, Role: FOLLOWER
I20260619 10:26:03.442909 11896 consensus_queue.cc:260] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.442945 11897 consensus_queue.cc:260] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.442994 11898 tablet_bootstrap.cc:492] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: No bootstrap required, opened a new log
I20260619 10:26:03.443032 11898 ts_tablet_manager.cc:1397] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:03.443066 11896 ts_tablet_manager.cc:1428] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:03.443114 11897 ts_tablet_manager.cc:1428] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:03.443121 11898 raft_consensus.cc:348] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.443176 11898 raft_consensus.cc:374] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:03.443192 11898 raft_consensus.cc:729] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c668b2331d41dabd1f098a62ccb646, State: Initialized, Role: FOLLOWER
I20260619 10:26:03.443251 11898 consensus_queue.cc:260] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.443400 11898 ts_tablet_manager.cc:1428] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:03.734283 11868 raft_consensus.cc:482] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:03.734385 11868 raft_consensus.cc:504] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.734526 11868 leader_election.cc:288] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843), d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563)
I20260619 10:26:03.734783 11597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbe18bc12f444b6ad537af41f3beead" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" is_pre_election: true
I20260619 10:26:03.734786 11456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbe18bc12f444b6ad537af41f3beead" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c532bbc77d4744bb207834b6c0f476" is_pre_election: true
I20260619 10:26:03.734882 11597 raft_consensus.cc:2454] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 0.
I20260619 10:26:03.734882 11456 raft_consensus.cc:2454] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 0.
I20260619 10:26:03.735031 11669 leader_election.cc:302] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c668b2331d41dabd1f098a62ccb646, d53dc9893c6b448f9f34ea5bfbae72b6; no voters:
I20260619 10:26:03.735137 11868 raft_consensus.cc:2781] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:03.735201 11868 raft_consensus.cc:482] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:03.735229 11868 raft_consensus.cc:3037] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:03.735900 11868 raft_consensus.cc:504] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.736027 11868 leader_election.cc:288] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 election: Requested vote from peers 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843), d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563)
I20260619 10:26:03.736169 11456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbe18bc12f444b6ad537af41f3beead" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c532bbc77d4744bb207834b6c0f476"
I20260619 10:26:03.736168 11597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbe18bc12f444b6ad537af41f3beead" candidate_uuid: "07c668b2331d41dabd1f098a62ccb646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6"
I20260619 10:26:03.736245 11456 raft_consensus.cc:3037] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:03.736267 11597 raft_consensus.cc:3037] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:03.736974 11456 raft_consensus.cc:2454] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 1.
I20260619 10:26:03.736974 11597 raft_consensus.cc:2454] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c668b2331d41dabd1f098a62ccb646 in term 1.
I20260619 10:26:03.737133 11670 leader_election.cc:302] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c668b2331d41dabd1f098a62ccb646, 19c532bbc77d4744bb207834b6c0f476; no voters:
I20260619 10:26:03.737268 11868 raft_consensus.cc:2781] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:03.737321 11868 raft_consensus.cc:686] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 1 LEADER]: Becoming Leader. State: Replica: 07c668b2331d41dabd1f098a62ccb646, State: Running, Role: LEADER
I20260619 10:26:03.737385 11868 consensus_queue.cc:237] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.737962 11299 catalog_manager.cc:5495] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67). New cstate: current_term: 1 leader_uuid: "07c668b2331d41dabd1f098a62ccb646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } health_report { overall_health: HEALTHY } } }
I20260619 10:26:03.797714 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.795526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.9.90.67:38687)
0619 10:26:03.795668 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:03.795670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:03.795676 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:26:03.795695 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:03.795695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:03.795703 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:03.795756 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:03.795894 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.796380 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.797099 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.797309 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.797362 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.797487 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:03.797521 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:03.797560 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:03.797613 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 10:26:03.797635 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260619 10:26:03.799716 11456 raft_consensus.cc:1261] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Refusing update from remote peer 07c668b2331d41dabd1f098a62ccb646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:03.799737 11597 raft_consensus.cc:1261] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Refusing update from remote peer 07c668b2331d41dabd1f098a62ccb646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:03.799932 11868 consensus_queue.cc:1035] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:03.800034 11863 consensus_queue.cc:1035] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:03.809259 11300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:03.809353 11300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:03.809902 11300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: Sending DeleteTablet for 3 replicas of tablet 7cbe18bc12f444b6ad537af41f3beead
I20260619 10:26:03.810101 11437 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cbe18bc12f444b6ad537af41f3beead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:03 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260619 10:26:03.810137 11577 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cbe18bc12f444b6ad537af41f3beead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:03 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46652
I20260619 10:26:03.810166 11717 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cbe18bc12f444b6ad537af41f3beead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:03 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:51998
I20260619 10:26:03.810242 11886 tablet_replica.cc:330] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: stopping tablet replica
I20260619 10:26:03.810246 11885 tablet_replica.cc:330] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: stopping tablet replica
I20260619 10:26:03.810282 11886 raft_consensus.cc:2229] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:03.810305 11885 raft_consensus.cc:2229] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:03.810323 11886 raft_consensus.cc:2258] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:03.810334 11885 raft_consensus.cc:2258] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:03.810397 11887 tablet_replica.cc:330] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: stopping tablet replica
I20260619 10:26:03.810439 11887 raft_consensus.cc:2229] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:03.810470 11887 raft_consensus.cc:2258] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:03.810578 11886 ts_tablet_manager.cc:1905] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:03.810796 11887 ts_tablet_manager.cc:1905] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:03.811914 11886 ts_tablet_manager.cc:1918] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:03.811941 11887 ts_tablet_manager.cc:1918] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:03.811971 11886 log.cc:1196] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/7cbe18bc12f444b6ad537af41f3beead
I20260619 10:26:03.811990 11887 log.cc:1196] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/7cbe18bc12f444b6ad537af41f3beead
I20260619 10:26:03.812039 11885 ts_tablet_manager.cc:1905] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:03.812160 11886 ts_tablet_manager.cc:1939] T 7cbe18bc12f444b6ad537af41f3beead P 07c668b2331d41dabd1f098a62ccb646: Deleting consensus metadata
I20260619 10:26:03.812179 11887 ts_tablet_manager.cc:1939] T 7cbe18bc12f444b6ad537af41f3beead P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting consensus metadata
I20260619 10:26:03.812407 11287 catalog_manager.cc:4841] TS 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67:38687): tablet 7cbe18bc12f444b6ad537af41f3beead (table test-table [id=4e0258f62f4745e697e91e795fd01883]) successfully deleted
I20260619 10:26:03.812429 11285 catalog_manager.cc:4841] TS d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563): tablet 7cbe18bc12f444b6ad537af41f3beead (table test-table [id=4e0258f62f4745e697e91e795fd01883]) successfully deleted
I20260619 10:26:03.812546 11300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:03.812670 11300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:03.813373 11885 ts_tablet_manager.cc:1918] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:03.813428 11885 log.cc:1196] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/7cbe18bc12f444b6ad537af41f3beead
I20260619 10:26:03.813611 11885 ts_tablet_manager.cc:1939] T 7cbe18bc12f444b6ad537af41f3beead P 19c532bbc77d4744bb207834b6c0f476: Deleting consensus metadata
I20260619 10:26:03.813748 11437 tablet_service.cc:1467] Processing CreateTablet for tablet e5a1455f51f84800bd4c36606aa55885 (DEFAULT_TABLE table=test-table [id=479fe601ffb44b7992ea50a07f03a214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:03.813773 11717 tablet_service.cc:1467] Processing CreateTablet for tablet e5a1455f51f84800bd4c36606aa55885 (DEFAULT_TABLE table=test-table [id=479fe601ffb44b7992ea50a07f03a214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:03.813858 11577 tablet_service.cc:1467] Processing CreateTablet for tablet e5a1455f51f84800bd4c36606aa55885 (DEFAULT_TABLE table=test-table [id=479fe601ffb44b7992ea50a07f03a214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:03.813869 11437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a1455f51f84800bd4c36606aa55885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:03.813886 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a1455f51f84800bd4c36606aa55885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:03.813896 11286 catalog_manager.cc:4841] TS 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843): tablet 7cbe18bc12f444b6ad537af41f3beead (table test-table [id=4e0258f62f4745e697e91e795fd01883]) successfully deleted
I20260619 10:26:03.813933 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a1455f51f84800bd4c36606aa55885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:03.814694 11897 tablet_bootstrap.cc:492] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476: Bootstrap starting.
I20260619 10:26:03.814970 11897 tablet_bootstrap.cc:654] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:03.814980 11898 tablet_bootstrap.cc:492] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: Bootstrap starting.
I20260619 10:26:03.815251 11898 tablet_bootstrap.cc:654] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:03.815481 11897 tablet_bootstrap.cc:492] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476: No bootstrap required, opened a new log
I20260619 10:26:03.815505 11896 tablet_bootstrap.cc:492] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: Bootstrap starting.
I20260619 10:26:03.815526 11897 ts_tablet_manager.cc:1397] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:03.815622 11897 raft_consensus.cc:348] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.815685 11897 raft_consensus.cc:374] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:03.815703 11897 raft_consensus.cc:729] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c532bbc77d4744bb207834b6c0f476, State: Initialized, Role: FOLLOWER
I20260619 10:26:03.815721 11898 tablet_bootstrap.cc:492] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: No bootstrap required, opened a new log
I20260619 10:26:03.815744 11897 consensus_queue.cc:260] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.815769 11898 ts_tablet_manager.cc:1397] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:03.815824 11896 tablet_bootstrap.cc:654] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:03.815845 11897 ts_tablet_manager.cc:1428] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:03.815869 11898 raft_consensus.cc:348] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.815927 11898 raft_consensus.cc:374] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:03.815943 11898 raft_consensus.cc:729] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c668b2331d41dabd1f098a62ccb646, State: Initialized, Role: FOLLOWER
I20260619 10:26:03.815986 11898 consensus_queue.cc:260] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.816058 11898 ts_tablet_manager.cc:1428] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:03.816607 11896 tablet_bootstrap.cc:492] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: No bootstrap required, opened a new log
I20260619 10:26:03.816649 11896 ts_tablet_manager.cc:1397] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:03.816782 11896 raft_consensus.cc:348] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.816852 11896 raft_consensus.cc:374] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:03.816869 11896 raft_consensus.cc:729] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53dc9893c6b448f9f34ea5bfbae72b6, State: Initialized, Role: FOLLOWER
I20260619 10:26:03.816915 11896 consensus_queue.cc:260] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.816998 11896 ts_tablet_manager.cc:1428] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:03.950481 11902 raft_consensus.cc:482] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:03.950587 11902 raft_consensus.cc:504] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.950906 11902 leader_election.cc:288] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843), 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67:38687)
I20260619 10:26:03.953893 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.951313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:39405 (local address 127.9.90.67:38687)
0619 10:26:03.951369 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:03.951370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:03.951378 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:03.951402 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:03.951403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:03.951411 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:03.951456 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:03.951609 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.952104 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.953184 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.953562 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.953623 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.953634 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:03.953772 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 10:26:03.953830 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:26:03.953902 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.951147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:38977 (local address 127.9.90.65:45843)
0619 10:26:03.951279 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:03.951281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:03.951347 (+ 66us) server_negotiation.cc:407] Connection header received
0619 10:26:03.951405 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:03.951406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:03.951418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:03.951476 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:03.951615 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.952098 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.953177 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.953561 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.953621 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.953633 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:03.953764 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 10:26:03.953825 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":49,"threads_started":1}
I20260619 10:26:03.953923 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.951098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:45843 (local address 127.9.90.66:38977)
0619 10:26:03.951281 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:03.951290 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:03.951351 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:03.951484 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:03.951484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:03.951487 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:03.951557 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:03.951558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.952199 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.952200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:03.953099 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:03.953102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.953633 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.953634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:03.953700 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.953707 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 10:26:03.953759 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:26:03.953821 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":26,"thread_start_us":109,"threads_started":1}
I20260619 10:26:03.953929 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.951092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38687 (local address 127.9.90.66:39405)
0619 10:26:03.951218 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:03.951229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:03.951289 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:03.951473 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:03.951474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:03.951476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:03.951548 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:03.951550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.952174 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.952175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:03.953098 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:03.953101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.953637 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.953638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:03.953700 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.953704 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:03.953748 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 10:26:03.953812 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":43,"thread_start_us":59,"threads_started":1}
I20260619 10:26:03.954142 11456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1455f51f84800bd4c36606aa55885" candidate_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c532bbc77d4744bb207834b6c0f476" is_pre_election: true
I20260619 10:26:03.954231 11456 raft_consensus.cc:2454] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d53dc9893c6b448f9f34ea5bfbae72b6 in term 0.
I20260619 10:26:03.954401 11530 leader_election.cc:302] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19c532bbc77d4744bb207834b6c0f476, d53dc9893c6b448f9f34ea5bfbae72b6; no voters:
I20260619 10:26:03.954447 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1455f51f84800bd4c36606aa55885" candidate_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c668b2331d41dabd1f098a62ccb646" is_pre_election: true
I20260619 10:26:03.954515 11737 raft_consensus.cc:2454] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d53dc9893c6b448f9f34ea5bfbae72b6 in term 0.
I20260619 10:26:03.954535 11902 raft_consensus.cc:2781] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:03.954574 11902 raft_consensus.cc:482] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:03.954593 11902 raft_consensus.cc:3037] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:03.955269 11902 raft_consensus.cc:504] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.955387 11902 leader_election.cc:288] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [CANDIDATE]: Term 1 election: Requested vote from peers 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843), 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67:38687)
I20260619 10:26:03.955528 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1455f51f84800bd4c36606aa55885" candidate_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c668b2331d41dabd1f098a62ccb646"
I20260619 10:26:03.955600 11737 raft_consensus.cc:3037] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:03.956152 11737 raft_consensus.cc:2454] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53dc9893c6b448f9f34ea5bfbae72b6 in term 1.
I20260619 10:26:03.956256 11456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1455f51f84800bd4c36606aa55885" candidate_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19c532bbc77d4744bb207834b6c0f476"
I20260619 10:26:03.956326 11456 raft_consensus.cc:3037] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:03.956336 11531 leader_election.cc:302] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c668b2331d41dabd1f098a62ccb646, d53dc9893c6b448f9f34ea5bfbae72b6; no voters:
I20260619 10:26:03.956423 11902 raft_consensus.cc:2781] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:03.956570 11902 raft_consensus.cc:686] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 LEADER]: Becoming Leader. State: Replica: d53dc9893c6b448f9f34ea5bfbae72b6, State: Running, Role: LEADER
I20260619 10:26:03.956638 11902 consensus_queue.cc:237] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } }
I20260619 10:26:03.957048 11456 raft_consensus.cc:2454] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53dc9893c6b448f9f34ea5bfbae72b6 in term 1.
I20260619 10:26:03.957304 11300 catalog_manager.cc:5495] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66). New cstate: current_term: 1 leader_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d53dc9893c6b448f9f34ea5bfbae72b6" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 } health_report { overall_health: UNKNOWN } } }
I20260619 10:26:03.963992 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.961953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.9.90.66:45563)
0619 10:26:03.962081 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:03.962082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:03.962088 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:26:03.962115 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:03.962115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:03.962130 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:03.962193 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:03.962274 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.962715 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.963392 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.963584 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.963635 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.963750 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:03.963844 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:03.963891 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:03.963908 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 10:26:03.963922 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":89,"threads_started":1}
I20260619 10:26:03.968518 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.966217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:50241 (local address 127.9.90.67:38687)
0619 10:26:03.966290 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:03.966292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:03.966647 (+ 355us) server_negotiation.cc:407] Connection header received
0619 10:26:03.966695 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:03.966696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:03.966706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:03.966749 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:03.966858 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.967285 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.968018 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.968189 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.968230 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.968366 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:03.968389 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:03.968419 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:03.968451 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 10:26:03.968467 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 10:26:03.968537 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.966139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38687 (local address 127.9.90.66:50241)
0619 10:26:03.966556 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:03.966583 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:03.966645 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:03.966759 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:03.966759 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:03.966761 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:03.966812 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:03.966813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.967392 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.967393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:03.967954 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:03.967955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.968252 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.968253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:03.968298 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.968311 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:03.968420 (+ 109us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:03.968421 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:26:03.968448 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 10:26:03.968477 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":345,"threads_started":1}
I20260619 10:26:03.970256 11456 raft_consensus.cc:1261] T e5a1455f51f84800bd4c36606aa55885 P 19c532bbc77d4744bb207834b6c0f476 [term 1 FOLLOWER]: Refusing update from remote peer d53dc9893c6b448f9f34ea5bfbae72b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:03.970271 11737 raft_consensus.cc:1261] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Refusing update from remote peer d53dc9893c6b448f9f34ea5bfbae72b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:03.970484 11916 consensus_queue.cc:1035] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c668b2331d41dabd1f098a62ccb646" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:03.970602 11902 consensus_queue.cc:1035] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19c532bbc77d4744bb207834b6c0f476" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:03.972075 11866 mvcc.cc:205] Tried to move back new op lower bound from 7298518073220837376 to 7298518073166831616. Current Snapshot: MvccSnapshot[applied={T|T < 7298518073220837376}]
I20260619 10:26:03.976807 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.974766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:59143 (local address 127.9.90.66:45563)
0619 10:26:03.974809 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:03.974811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:03.974831 (+ 20us) server_negotiation.cc:407] Connection header received
0619 10:26:03.974883 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:03.974883 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:03.974893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:03.974937 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:03.975048 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.975487 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.976194 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.976421 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.976466 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.976602 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:03.976635 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:03.976679 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:03.976726 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 10:26:03.976746 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:26:03.976840 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:03.974650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45563 (local address 127.9.90.67:59143)
0619 10:26:03.974779 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:03.974789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:03.974829 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:03.974943 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:03.974944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:03.974946 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:03.974988 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:03.974990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.975539 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.975539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:03.976073 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:03.976075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:03.976482 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:03.976483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:03.976535 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:03.976541 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:03.976684 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:03.976686 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:26:03.976725 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 10:26:03.976765 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":84,"threads_started":1}
W20260619 10:26:03.977075 11869 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 10:26:05.493805 11299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:05.493903 11299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:05.494624 11299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d91ff395ffb4cb5bac39343ad403c79: Sending DeleteTablet for 3 replicas of tablet e5a1455f51f84800bd4c36606aa55885
I20260619 10:26:05.494853 11577 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a1455f51f84800bd4c36606aa55885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:05 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46652
I20260619 10:26:05.494854 11437 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a1455f51f84800bd4c36606aa55885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:05 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260619 10:26:05.494889 11717 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a1455f51f84800bd4c36606aa55885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:05 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:51998
I20260619 10:26:05.494975 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 11372
I20260619 10:26:05.497912 11951 tablet_replica.cc:330] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: stopping tablet replica
I20260619 10:26:05.497973 11951 raft_consensus.cc:2229] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:05.498051 11951 raft_consensus.cc:2258] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:05.498155 11952 tablet_replica.cc:330] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: stopping tablet replica
I20260619 10:26:05.498210 11952 raft_consensus.cc:2229] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:05.498248 11952 raft_consensus.cc:2258] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:05.498347 11951 ts_tablet_manager.cc:1905] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:05.498473 11952 ts_tablet_manager.cc:1905] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:05.499982 11951 ts_tablet_manager.cc:1918] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:26:05.499975 11952 ts_tablet_manager.cc:1918] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 10:26:05.500036 11952 log.cc:1196] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/e5a1455f51f84800bd4c36606aa55885
I20260619 10:26:05.500036 11951 log.cc:1196] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/e5a1455f51f84800bd4c36606aa55885
I20260619 10:26:05.500231 11952 ts_tablet_manager.cc:1939] T e5a1455f51f84800bd4c36606aa55885 P 07c668b2331d41dabd1f098a62ccb646: Deleting consensus metadata
I20260619 10:26:05.500258 11951 ts_tablet_manager.cc:1939] T e5a1455f51f84800bd4c36606aa55885 P d53dc9893c6b448f9f34ea5bfbae72b6: Deleting consensus metadata
I20260619 10:26:05.501201 11285 catalog_manager.cc:4841] TS d53dc9893c6b448f9f34ea5bfbae72b6 (127.9.90.66:45563): tablet e5a1455f51f84800bd4c36606aa55885 (table test-table [id=479fe601ffb44b7992ea50a07f03a214]) successfully deleted
I20260619 10:26:05.501255 11287 catalog_manager.cc:4841] TS 07c668b2331d41dabd1f098a62ccb646 (127.9.90.67:38687): tablet e5a1455f51f84800bd4c36606aa55885 (table test-table [id=479fe601ffb44b7992ea50a07f03a214]) successfully deleted
W20260619 10:26:05.501873 11286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:45843 (error 108)
I20260619 10:26:05.502077 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 11512
I20260619 10:26:05.505944 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:05.502246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:45843 (local address 127.0.0.1:36000)
0619 10:26:05.505867 (+ 3621us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:05.505912 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3599,"thread_start_us":61,"threads_started":1}
W20260619 10:26:05.506057 11286 catalog_manager.cc:4584] TS 19c532bbc77d4744bb207834b6c0f476 (127.9.90.65:45843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5a1455f51f84800bd4c36606aa55885: Network error: Client connection negotiation failed: client connection to 127.9.90.65:45843: connect: Connection refused (error 111)
I20260619 10:26:05.507412 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 11652
I20260619 10:26:05.512168 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 11268
2026-06-19T10:26:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5589 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:05 dist-test-slave-hw7q krb5kdc[11958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:05 dist-test-slave-hw7q krb5kdc[11958](info): set up 2 sockets
Jun 19 10:26:05 dist-test-slave-hw7q krb5kdc[11958](info): commencing operation
krb5kdc: starting...
W20260619 10:26:06.473325 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:06 dist-test-slave-hw7q krb5kdc[11958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:06.576916 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40593
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:06.653014 11971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:06.653208 11971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:06.653237 11971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:06.654386 11971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:06.654426 11971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:06.654445 11971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:06.654459 11971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:06.654476 11971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:06.655890 11971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:40593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:06.656116 11971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:06.656309 11971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:06.660663 11979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:06.660665 11976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:06.660701 11977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:06.660903 11971 server_base.cc:1034] running on GCE node
I20260619 10:26:06.661116 11971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:06.661383 11971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:06.662524 11971 hybrid_clock.cc:648] HybridClock initialized: now 1781864766662513 us; error 26 us; skew 500 ppm
Jun 19 10:26:06 dist-test-slave-hw7q krb5kdc[11958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:06.664736 11971 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:06.665141 11971 webserver.cc:460] Webserver started at http://127.9.90.126:37183/ using document root <none> and password file <none>
I20260619 10:26:06.665360 11971 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:06.665405 11971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:06.665526 11971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:06.666549 11971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9"
format_stamp: "Formatted at 2026-06-19 10:26:06 on dist-test-slave-hw7q"
server_key: "12b46a323e592fb126b6c5434d50833e"
server_key_iv: "3d792258c3038570c55d6ba84a2cd8d0"
server_key_version: "encryptionkey@0"
I20260619 10:26:06.666855 11971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9"
format_stamp: "Formatted at 2026-06-19 10:26:06 on dist-test-slave-hw7q"
server_key: "12b46a323e592fb126b6c5434d50833e"
server_key_iv: "3d792258c3038570c55d6ba84a2cd8d0"
server_key_version: "encryptionkey@0"
I20260619 10:26:06.668016 11971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:06.668676 11986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:06.668839 11971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:06.668917 11971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9"
format_stamp: "Formatted at 2026-06-19 10:26:06 on dist-test-slave-hw7q"
server_key: "12b46a323e592fb126b6c5434d50833e"
server_key_iv: "3d792258c3038570c55d6ba84a2cd8d0"
server_key_version: "encryptionkey@0"
I20260619 10:26:06.668977 11971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:06.677584 11971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:06.679229 11971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:06.679361 11971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:06.682926 11971 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:40593
I20260619 10:26:06.682960 12038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:40593 every 8 connection(s)
I20260619 10:26:06.683254 11971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:06.684031 12039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:06.686177 12039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9: Bootstrap starting.
I20260619 10:26:06.686708 12039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:06.686937 12039 log.cc:824] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:06.687502 12039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9: No bootstrap required, opened a new log
I20260619 10:26:06.689015 12039 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40593 } }
I20260619 10:26:06.689146 12039 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:06.689168 12039 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd3d8010af846f8a9bb5c8f05d7d1b9, State: Initialized, Role: FOLLOWER
I20260619 10:26:06.689273 12039 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40593 } }
I20260619 10:26:06.689321 12039 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:06.689340 12039 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:06.689370 12039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:06.689852 12039 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40593 } }
I20260619 10:26:06.689940 12039 leader_election.cc:302] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfd3d8010af846f8a9bb5c8f05d7d1b9; no voters:
I20260619 10:26:06.690063 12039 leader_election.cc:288] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:06.690119 12042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:06.690267 12039 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:06.690455 12042 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [term 1 LEADER]: Becoming Leader. State: Replica: bfd3d8010af846f8a9bb5c8f05d7d1b9, State: Running, Role: LEADER
I20260619 10:26:06.690475 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 11971
I20260619 10:26:06.690552 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:06.690544 12042 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40593 } }
I20260619 10:26:06.690709 9577 external_mini_cluster.cc:1428] Setting key 389e40181473059b0c9cef69677aa914
I20260619 10:26:06.690869 12043 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40593 } } }
I20260619 10:26:06.690943 12043 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:06.691365 12044 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfd3d8010af846f8a9bb5c8f05d7d1b9. Latest consensus state: current_term: 1 leader_uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd3d8010af846f8a9bb5c8f05d7d1b9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40593 } } }
I20260619 10:26:06.691396 12046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:06.691497 12044 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:06.691984 12046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 10:26:06 dist-test-slave-hw7q krb5kdc[11958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:06.698143 12046 catalog_manager.cc:1347] Generated new cluster ID: 7ae966442f614c6a915199605a02175e
I20260619 10:26:06.698186 12046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:06.701675 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:06.691312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46650 (local address 127.9.90.126:40593)
0619 10:26:06.691466 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:06.691476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:06.691665 (+ 189us) server_negotiation.cc:407] Connection header received
0619 10:26:06.691815 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:06.691823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:06.691888 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:06.692888 (+ 1000us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:06.692979 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:06.694344 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:06.695078 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:06.695703 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:06.695818 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:06.698256 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:06.698260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:06.698261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:06.698272 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:06.700131 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:06.700489 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:06.700492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:06.700492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:06.700614 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:06.700791 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:06.700793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:06.700793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:06.701093 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:06.701245 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:06.701326 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 10:26:06.701462 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260619 10:26:06.703284 12046 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:06.703740 12046 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:06.708010 12046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfd3d8010af846f8a9bb5c8f05d7d1b9: Generated new TSK 0
I20260619 10:26:06.708149 12046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:26:06.730160 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40593
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:06.808394 12065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:06.808588 12065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:06.808609 12065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:06.809729 12065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:06.809769 12065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:06.809818 12065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:06.811203 12065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:06.811434 12065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:06.811661 12065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:06.816843 12070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:06.816840 12073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:06.816885 12071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:06.817173 12065 server_base.cc:1034] running on GCE node
I20260619 10:26:06.817382 12065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:06.817591 12065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:06.818723 12065 hybrid_clock.cc:648] HybridClock initialized: now 1781864766818711 us; error 24 us; skew 500 ppm
Jun 19 10:26:06 dist-test-slave-hw7q krb5kdc[11958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:06.820946 12065 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:26:06.821341 12065 webserver.cc:460] Webserver started at http://127.9.90.65:40771/ using document root <none> and password file <none>
I20260619 10:26:06.821543 12065 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:06.821589 12065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:06.821708 12065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:06.822728 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "3c828919b6ec42f7ae53642f6ab405f2"
format_stamp: "Formatted at 2026-06-19 10:26:06 on dist-test-slave-hw7q"
server_key: "2f9a9e0c3c0352c1a695803ba5a2aa4c"
server_key_iv: "8916f935e926ca5e155d986ed5dfdb1f"
server_key_version: "encryptionkey@0"
I20260619 10:26:06.823041 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "3c828919b6ec42f7ae53642f6ab405f2"
format_stamp: "Formatted at 2026-06-19 10:26:06 on dist-test-slave-hw7q"
server_key: "2f9a9e0c3c0352c1a695803ba5a2aa4c"
server_key_iv: "8916f935e926ca5e155d986ed5dfdb1f"
server_key_version: "encryptionkey@0"
I20260619 10:26:06.824179 12065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:06.824802 12080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:06.824939 12065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:06.825016 12065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "3c828919b6ec42f7ae53642f6ab405f2"
format_stamp: "Formatted at 2026-06-19 10:26:06 on dist-test-slave-hw7q"
server_key: "2f9a9e0c3c0352c1a695803ba5a2aa4c"
server_key_iv: "8916f935e926ca5e155d986ed5dfdb1f"
server_key_version: "encryptionkey@0"
I20260619 10:26:06.825071 12065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:06.854023 12065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:06.855737 12065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:06.855870 12065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:06.856073 12065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:06.856364 12065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:06.856398 12065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:06.856426 12065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:06.856451 12065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:06.861883 12065 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:36245
I20260619 10:26:06.861935 12193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:36245 every 8 connection(s)
I20260619 10:26:06.862221 12065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:06.864470 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 12065
I20260619 10:26:06.864554 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:06.864702 9577 external_mini_cluster.cc:1428] Setting key 05b0b426162978eb8cbfaa118f888066
Jun 19 10:26:06 dist-test-slave-hw7q krb5kdc[11958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:06.871702 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:06.862870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:37367 (local address 127.9.90.126:40593)
0619 10:26:06.862916 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:06.862918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:06.863125 (+ 207us) server_negotiation.cc:407] Connection header received
0619 10:26:06.863585 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:06.863587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:06.863599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:06.863642 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:06.864507 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:06.865217 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:06.866617 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:06.866929 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:06.867003 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:06.869125 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:06.869126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:06.869127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:06.869133 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:06.870392 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:06.870830 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:06.870830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:06.870831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:06.870875 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:06.871109 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:06.871110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:06.871110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:06.871199 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:06.871241 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:06.871574 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 10:26:06.871639 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:06.871937 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:06.862859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40593 (local address 127.9.90.65:37367)
0619 10:26:06.863013 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:06.863048 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:06.863513 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:06.863703 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:06.863706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:06.864218 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:06.864411 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:06.864415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:06.865311 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:06.865312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:06.866506 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:06.866509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:06.867014 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:06.867014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:06.867114 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:06.867572 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:06.867580 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:06.869040 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:06.870465 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:06.870471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:06.870473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:06.870764 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:06.870927 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:06.870928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:06.870928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:06.871053 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:06.871249 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:06.871255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:06.871440 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 10:26:06.871548 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 10:26:06.871712 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
I20260619 10:26:06.872443 12194 heartbeater.cc:347] Connected to a master server at 127.9.90.126:40593
I20260619 10:26:06.872535 12194 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:06.872699 12194 heartbeater.cc:511] Master 127.9.90.126:40593 requested a full tablet report, sending...
I20260619 10:26:06.873278 12003 ts_manager.cc:194] Registered new tserver with Master: 3c828919b6ec42f7ae53642f6ab405f2 (127.9.90.65:36245)
I20260619 10:26:06.873924 12003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:37367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:26:06.885644 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40593
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:06.962909 12199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:06.963100 12199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:06.963121 12199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:06.964196 12199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:06.964229 12199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:06.964253 12199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:06.965571 12199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:06.965752 12199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:06.965952 12199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:06.970548 12205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:06.970541 12204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:06.970541 12207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:06.970897 12199 server_base.cc:1034] running on GCE node
I20260619 10:26:06.971063 12199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:06.971251 12199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:06.972374 12199 hybrid_clock.cc:648] HybridClock initialized: now 1781864766972344 us; error 45 us; skew 500 ppm
Jun 19 10:26:06 dist-test-slave-hw7q krb5kdc[11958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:06.974637 12199 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:26:06.975049 12199 webserver.cc:460] Webserver started at http://127.9.90.66:44559/ using document root <none> and password file <none>
I20260619 10:26:06.975237 12199 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:06.975278 12199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:06.975396 12199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:06.976384 12199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "cb3f33df6bf04fe49054fa2cf3a40929"
format_stamp: "Formatted at 2026-06-19 10:26:06 on dist-test-slave-hw7q"
server_key: "d3ba38d6aff4e0657c0ccf6440a7773a"
server_key_iv: "0b4edce0c541d35a8dd465c95d3f3c5a"
server_key_version: "encryptionkey@0"
I20260619 10:26:06.976699 12199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "cb3f33df6bf04fe49054fa2cf3a40929"
format_stamp: "Formatted at 2026-06-19 10:26:06 on dist-test-slave-hw7q"
server_key: "d3ba38d6aff4e0657c0ccf6440a7773a"
server_key_iv: "0b4edce0c541d35a8dd465c95d3f3c5a"
server_key_version: "encryptionkey@0"
I20260619 10:26:06.977887 12199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:06.978513 12214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:06.978650 12199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:06.978729 12199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "cb3f33df6bf04fe49054fa2cf3a40929"
format_stamp: "Formatted at 2026-06-19 10:26:06 on dist-test-slave-hw7q"
server_key: "d3ba38d6aff4e0657c0ccf6440a7773a"
server_key_iv: "0b4edce0c541d35a8dd465c95d3f3c5a"
server_key_version: "encryptionkey@0"
I20260619 10:26:06.978788 12199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:06.990345 12199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:06.992012 12199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:06.992133 12199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:06.992313 12199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:06.992605 12199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:06.992640 12199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:06.992666 12199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:06.992689 12199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:06.998140 12199 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:39369
I20260619 10:26:06.998173 12327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:39369 every 8 connection(s)
I20260619 10:26:06.998453 12199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:06.999528 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 12199
I20260619 10:26:06.999620 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:06.999783 9577 external_mini_cluster.cc:1428] Setting key f99012fc85deca4f5626e54e6a8d5d10
Jun 19 10:26:07 dist-test-slave-hw7q krb5kdc[11958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
I20260619 10:26:07.008312 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:06.999047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:59269 (local address 127.9.90.126:40593)
0619 10:26:06.999089 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:06.999091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:06.999279 (+ 188us) server_negotiation.cc:407] Connection header received
0619 10:26:06.999883 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:06.999884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:06.999895 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:06.999969 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:07.000987 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:07.001483 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:07.002532 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:07.002754 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:07.002816 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:07.005076 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:07.005077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:07.005078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:07.005085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:07.006733 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:07.007197 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:07.007198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:07.007199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:07.007270 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:07.007511 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:07.007512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:07.007512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:07.007658 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:07.007712 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:07.008052 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 10:26:07.008191 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:07.008391 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:06.999023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40593 (local address 127.9.90.66:59269)
0619 10:26:06.999165 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:06.999203 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:06.999788 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:07.000029 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:07.000033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:07.000628 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:07.000893 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:07.000901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:07.001575 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:07.001576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:07.002460 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:07.002463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:07.002840 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:07.002841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:07.002975 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:07.003432 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:07.003441 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:07.004975 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:07.006828 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:07.006834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:07.006836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:07.007118 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:07.007325 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:07.007326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:07.007327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:07.007460 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:07.007741 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:07.007747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:07.007913 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 10:26:07.008023 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 10:26:07.008176 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260619 10:26:07.009019 12328 heartbeater.cc:347] Connected to a master server at 127.9.90.126:40593
I20260619 10:26:07.009121 12328 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:07.009307 12328 heartbeater.cc:511] Master 127.9.90.126:40593 requested a full tablet report, sending...
I20260619 10:26:07.009706 12003 ts_manager.cc:194] Registered new tserver with Master: cb3f33df6bf04fe49054fa2cf3a40929 (127.9.90.66:39369)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
I20260619 10:26:07.010247 12003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:59269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:26:07.022136 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40593
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:07.101378 12333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:07.101578 12333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:07.101608 12333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:07.102775 12333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:07.102815 12333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:07.102850 12333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:07.104240 12333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:07.104465 12333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:07.104657 12333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:07.109267 12341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:07.109246 12339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:07.109241 12338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:07.109302 12333 server_base.cc:1034] running on GCE node
I20260619 10:26:07.109579 12333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:07.109839 12333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:07.111006 12333 hybrid_clock.cc:648] HybridClock initialized: now 1781864767110991 us; error 30 us; skew 500 ppm
Jun 19 10:26:07 dist-test-slave-hw7q krb5kdc[11958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864767, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:07.113384 12333 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:26:07.113832 12333 webserver.cc:460] Webserver started at http://127.9.90.67:38455/ using document root <none> and password file <none>
I20260619 10:26:07.114027 12333 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:07.114073 12333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:07.114189 12333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:07.115157 12333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "b732a8511f2c4a8abb3eb9aa5cee75b5"
format_stamp: "Formatted at 2026-06-19 10:26:07 on dist-test-slave-hw7q"
server_key: "0a6def8ee77c5e17d53471c218ea8347"
server_key_iv: "f5a0d2cdfb924d68803befe869f5b375"
server_key_version: "encryptionkey@0"
I20260619 10:26:07.115476 12333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "b732a8511f2c4a8abb3eb9aa5cee75b5"
format_stamp: "Formatted at 2026-06-19 10:26:07 on dist-test-slave-hw7q"
server_key: "0a6def8ee77c5e17d53471c218ea8347"
server_key_iv: "f5a0d2cdfb924d68803befe869f5b375"
server_key_version: "encryptionkey@0"
I20260619 10:26:07.116626 12333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:07.117240 12348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:07.117391 12333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:07.117467 12333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "b732a8511f2c4a8abb3eb9aa5cee75b5"
format_stamp: "Formatted at 2026-06-19 10:26:07 on dist-test-slave-hw7q"
server_key: "0a6def8ee77c5e17d53471c218ea8347"
server_key_iv: "f5a0d2cdfb924d68803befe869f5b375"
server_key_version: "encryptionkey@0"
I20260619 10:26:07.117525 12333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:07.148221 12333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:07.150126 12333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:07.150267 12333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:07.150487 12333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:07.150805 12333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:07.150839 12333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:07.150867 12333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:07.150884 12333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:07.155994 12333 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:33451
I20260619 10:26:07.156054 12461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:33451 every 8 connection(s)
I20260619 10:26:07.156327 12333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:26:07.156440 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 12333
I20260619 10:26:07.156533 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:07.156756 9577 external_mini_cluster.cc:1428] Setting key 2047c5a4cd56743dff1e5be832c0a96d
Jun 19 10:26:07 dist-test-slave-hw7q krb5kdc[11958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864767, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:07.164588 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:07.157036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:55903 (local address 127.9.90.126:40593)
0619 10:26:07.157088 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:07.157090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:07.157299 (+ 209us) server_negotiation.cc:407] Connection header received
0619 10:26:07.157902 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:07.157903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:07.157913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:07.157956 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:07.158831 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:07.159411 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:07.160344 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:07.160522 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:07.160574 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:07.162518 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:07.162519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:07.162520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:07.162526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:07.163608 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:07.163925 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:07.163926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:07.163926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:07.163967 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:07.164142 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:07.164143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:07.164143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:07.164233 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:07.164289 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:07.164471 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 10:26:07.164527 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:26:07.164770 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:07.157040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40593 (local address 127.9.90.67:55903)
0619 10:26:07.157183 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:07.157229 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:07.157825 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:07.158014 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:07.158018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:07.158539 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:07.158738 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:07.158743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:07.159485 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:07.159486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:07.160270 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:07.160273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:07.160581 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:07.160581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:07.160684 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:07.161134 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:07.161143 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:07.162444 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:07.163670 (+ 1226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:07.163675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:07.163677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:07.163871 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:07.164009 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:07.164009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:07.164009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:07.164093 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:07.164274 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:07.164276 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:07.164381 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 10:26:07.164454 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 10:26:07.164560 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260619 10:26:07.165232 12462 heartbeater.cc:347] Connected to a master server at 127.9.90.126:40593
I20260619 10:26:07.165309 12462 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:07.165439 12462 heartbeater.cc:511] Master 127.9.90.126:40593 requested a full tablet report, sending...
I20260619 10:26:07.165731 12003 ts_manager.cc:194] Registered new tserver with Master: b732a8511f2c4a8abb3eb9aa5cee75b5 (127.9.90.67:33451)
I20260619 10:26:07.166152 12003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:55903
I20260619 10:26:07.168506 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 10:26:07.178211 12052 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:07.177382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.9.90.126:40593)
0619 10:26:07.177437 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:07.177439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:07.177528 (+ 89us) server_negotiation.cc:407] Connection header received
0619 10:26:07.177575 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:07.177576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:07.177584 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:07.177632 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:07.178147 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46666: BlockingRecv error: recv got EOF from 127.0.0.1:46666 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:26:07.179060 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 12065
I20260619 10:26:07.182839 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 12199
I20260619 10:26:07.187094 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 12333
I20260619 10:26:07.190514 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 11971
2026-06-19T10:26:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1677 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:07 dist-test-slave-hw7q krb5kdc[12474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:07 dist-test-slave-hw7q krb5kdc[12474](info): set up 2 sockets
Jun 19 10:26:07 dist-test-slave-hw7q krb5kdc[12474](info): commencing operation
krb5kdc: starting...
W20260619 10:26:08.141361 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:08 dist-test-slave-hw7q krb5kdc[12474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:08.244012 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:37861
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:37861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:08.320919 12487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:08.321118 12487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:08.321147 12487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:08.322335 12487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:08.322372 12487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:08.322391 12487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:08.322407 12487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:08.322423 12487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:08.323875 12487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:37861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:08.324105 12487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:08.324303 12487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:08.328801 12495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:08.328807 12493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:08.328967 12487 server_base.cc:1034] running on GCE node
W20260619 10:26:08.328824 12492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:08.329239 12487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:08.329526 12487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:08.330659 12487 hybrid_clock.cc:648] HybridClock initialized: now 1781864768330647 us; error 30 us; skew 500 ppm
Jun 19 10:26:08 dist-test-slave-hw7q krb5kdc[12474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864768, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:08.332973 12487 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:08.333362 12487 webserver.cc:460] Webserver started at http://127.9.90.126:33095/ using document root <none> and password file <none>
I20260619 10:26:08.333525 12487 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:08.333560 12487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:08.333648 12487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:08.334635 12487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "3e1b3a5bc1214ae9a611322cc622c262"
format_stamp: "Formatted at 2026-06-19 10:26:08 on dist-test-slave-hw7q"
server_key: "d47a32a9a94e647c491f66d1b56c4011"
server_key_iv: "1d85b0cab5632e3b9396a0c745589104"
server_key_version: "encryptionkey@0"
I20260619 10:26:08.334914 12487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "3e1b3a5bc1214ae9a611322cc622c262"
format_stamp: "Formatted at 2026-06-19 10:26:08 on dist-test-slave-hw7q"
server_key: "d47a32a9a94e647c491f66d1b56c4011"
server_key_iv: "1d85b0cab5632e3b9396a0c745589104"
server_key_version: "encryptionkey@0"
I20260619 10:26:08.336024 12487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:08.336621 12502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:08.336756 12487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:08.336827 12487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "3e1b3a5bc1214ae9a611322cc622c262"
format_stamp: "Formatted at 2026-06-19 10:26:08 on dist-test-slave-hw7q"
server_key: "d47a32a9a94e647c491f66d1b56c4011"
server_key_iv: "1d85b0cab5632e3b9396a0c745589104"
server_key_version: "encryptionkey@0"
I20260619 10:26:08.336894 12487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:08.355333 12487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:08.356985 12487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:08.357132 12487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:08.360883 12487 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:37861
I20260619 10:26:08.360918 12554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:37861 every 8 connection(s)
I20260619 10:26:08.361248 12487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:08.362123 12555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:08.364260 12555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262: Bootstrap starting.
I20260619 10:26:08.364804 12555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:08.365049 12555 log.cc:824] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:08.365669 12555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262: No bootstrap required, opened a new log
I20260619 10:26:08.367192 12555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b3a5bc1214ae9a611322cc622c262" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37861 } }
I20260619 10:26:08.367359 12555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:08.367406 12555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1b3a5bc1214ae9a611322cc622c262, State: Initialized, Role: FOLLOWER
I20260619 10:26:08.367545 12555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b3a5bc1214ae9a611322cc622c262" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37861 } }
I20260619 10:26:08.367623 12555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:08.367659 12555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:08.367704 12555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:08.367942 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 12487
I20260619 10:26:08.368018 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:08.368162 9577 external_mini_cluster.cc:1428] Setting key fe50188383644e5663354cfb9f466a3b
I20260619 10:26:08.368507 12555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b3a5bc1214ae9a611322cc622c262" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37861 } }
I20260619 10:26:08.368606 12555 leader_election.cc:302] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e1b3a5bc1214ae9a611322cc622c262; no voters:
I20260619 10:26:08.368734 12555 leader_election.cc:288] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:08.368850 12560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:08.368969 12555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:08.368983 12560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [term 1 LEADER]: Becoming Leader. State: Replica: 3e1b3a5bc1214ae9a611322cc622c262, State: Running, Role: LEADER
I20260619 10:26:08.369163 12560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b3a5bc1214ae9a611322cc622c262" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37861 } }
I20260619 10:26:08.369526 12562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1b3a5bc1214ae9a611322cc622c262. Latest consensus state: current_term: 1 leader_uuid: "3e1b3a5bc1214ae9a611322cc622c262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b3a5bc1214ae9a611322cc622c262" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37861 } } }
I20260619 10:26:08.369535 12561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e1b3a5bc1214ae9a611322cc622c262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b3a5bc1214ae9a611322cc622c262" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37861 } } }
I20260619 10:26:08.369608 12562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:08.369632 12561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:08.370059 12569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:08.370556 12569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:08.372289 12569 catalog_manager.cc:1347] Generated new cluster ID: 755e10ced5b7464ba121b0d57e82ee23
I20260619 10:26:08.372340 12569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:26:08 dist-test-slave-hw7q krb5kdc[12474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:08.379554 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:08.368576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.9.90.126:37861)
0619 10:26:08.368753 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:08.368762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:08.368813 (+ 51us) server_negotiation.cc:407] Connection header received
0619 10:26:08.368932 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:08.368940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:08.369001 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:08.369768 (+ 767us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:08.369965 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:08.371398 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:08.372240 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:08.372806 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:08.372906 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:08.375523 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:08.375527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:08.375528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:08.375540 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:08.377998 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:08.378397 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:08.378400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:08.378400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:08.378507 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:08.378672 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:08.378673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:08.378673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:08.378991 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:08.379143 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:08.379211 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 10:26:08.379351 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260619 10:26:08.380388 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:26:08.384132 12569 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:08.384747 12569 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:08.389990 12569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e1b3a5bc1214ae9a611322cc622c262: Generated new TSK 0
I20260619 10:26:08.390146 12569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:26:08.395474 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:08.392025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.9.90.126:37861)
0619 10:26:08.392063 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:08.392065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:08.392169 (+ 104us) server_negotiation.cc:407] Connection header received
0619 10:26:08.392208 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:08.392208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:08.392220 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:08.392286 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:08.392582 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:08.393351 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:08.394070 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:08.394415 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:08.394478 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:08.395014 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:08.395015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:08.395015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:08.395220 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:08.395237 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:08.395245 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:08.395304 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:08.395377 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 10:26:08.395399 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:26:08.397625 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 12487
2026-06-19T10:26:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1205 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:08 dist-test-slave-hw7q krb5kdc[12588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:08 dist-test-slave-hw7q krb5kdc[12588](info): set up 2 sockets
Jun 19 10:26:08 dist-test-slave-hw7q krb5kdc[12588](info): commencing operation
krb5kdc: starting...
W20260619 10:26:09.345232 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:09 dist-test-slave-hw7q krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:09.449941 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42571
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:09.526808 12601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:09.526999 12601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:09.527019 12601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:09.528133 12601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:09.528174 12601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:09.528187 12601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:09.528198 12601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:09.528216 12601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:09.529655 12601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:42571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:09.529891 12601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:09.530088 12601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:09.534446 12606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:09.534485 12607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:09.534638 12601 server_base.cc:1034] running on GCE node
W20260619 10:26:09.534493 12609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:09.534935 12601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:09.535197 12601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:09.536365 12601 hybrid_clock.cc:648] HybridClock initialized: now 1781864769536349 us; error 28 us; skew 500 ppm
Jun 19 10:26:09 dist-test-slave-hw7q krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864769, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:09.538951 12601 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:09.539368 12601 webserver.cc:460] Webserver started at http://127.9.90.126:39299/ using document root <none> and password file <none>
I20260619 10:26:09.539556 12601 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:09.539601 12601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:09.539722 12601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:09.540732 12601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "1335c44e070748e4b60b4dde350354fe"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "dc3d92b08e75ad6a2e27258edbff3f12"
server_key_iv: "c754deacb2cf304464792adbe9bded8d"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.541046 12601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "1335c44e070748e4b60b4dde350354fe"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "dc3d92b08e75ad6a2e27258edbff3f12"
server_key_iv: "c754deacb2cf304464792adbe9bded8d"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.542344 12601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:09.542963 12616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:09.543104 12601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:09.543182 12601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "1335c44e070748e4b60b4dde350354fe"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "dc3d92b08e75ad6a2e27258edbff3f12"
server_key_iv: "c754deacb2cf304464792adbe9bded8d"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.543252 12601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:09.550742 12601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:09.552348 12601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:09.552464 12601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:09.556195 12601 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:42571
I20260619 10:26:09.556224 12668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:42571 every 8 connection(s)
I20260619 10:26:09.556514 12601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:09.557412 12669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:09.559556 12669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe: Bootstrap starting.
I20260619 10:26:09.560056 12669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:09.560302 12669 log.cc:824] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:09.560906 12669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe: No bootstrap required, opened a new log
I20260619 10:26:09.562482 12669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1335c44e070748e4b60b4dde350354fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42571 } }
I20260619 10:26:09.562613 12669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:09.562636 12669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1335c44e070748e4b60b4dde350354fe, State: Initialized, Role: FOLLOWER
I20260619 10:26:09.562765 12669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1335c44e070748e4b60b4dde350354fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42571 } }
I20260619 10:26:09.562839 12669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:09.562872 12669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:09.562917 12669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:09.563683 12669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1335c44e070748e4b60b4dde350354fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42571 } }
I20260619 10:26:09.563787 12669 leader_election.cc:302] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1335c44e070748e4b60b4dde350354fe; no voters:
I20260619 10:26:09.563827 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 12601
I20260619 10:26:09.563925 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:09.563941 12669 leader_election.cc:288] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:09.563962 12672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:09.564077 12672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [term 1 LEADER]: Becoming Leader. State: Replica: 1335c44e070748e4b60b4dde350354fe, State: Running, Role: LEADER
I20260619 10:26:09.564092 9577 external_mini_cluster.cc:1428] Setting key f617b89aa45f8740040d0fa4f1d51538
I20260619 10:26:09.564167 12669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:09.564154 12672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1335c44e070748e4b60b4dde350354fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42571 } }
I20260619 10:26:09.564548 12674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1335c44e070748e4b60b4dde350354fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1335c44e070748e4b60b4dde350354fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42571 } } }
I20260619 10:26:09.564538 12673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1335c44e070748e4b60b4dde350354fe. Latest consensus state: current_term: 1 leader_uuid: "1335c44e070748e4b60b4dde350354fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1335c44e070748e4b60b4dde350354fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42571 } } }
I20260619 10:26:09.564673 12674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:09.564702 12673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:09.564849 12678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:09.565327 12678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:09.566988 12678 catalog_manager.cc:1347] Generated new cluster ID: b42ff11390f74765840117f5dabdec97
I20260619 10:26:09.567039 12678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:26:09 dist-test-slave-hw7q krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:09.573321 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:09.564545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.9.90.126:42571)
0619 10:26:09.564707 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:09.564717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:09.564964 (+ 247us) server_negotiation.cc:407] Connection header received
0619 10:26:09.565078 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:09.565085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:09.565139 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:09.565269 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:09.566144 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.567305 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.568018 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.568420 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.568511 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:09.570590 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:09.570595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:09.570595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:09.570606 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:09.572012 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:09.572313 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:09.572316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:09.572317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:09.572402 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:09.572570 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:09.572571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:09.572571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:09.572782 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:09.572923 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:09.573013 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 10:26:09.573141 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260619 10:26:09.586496 12678 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:09.586966 12678 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:09.590669 12678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1335c44e070748e4b60b4dde350354fe: Generated new TSK 0
I20260619 10:26:09.590849 12678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:26:09.615823 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42571
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:09.693645 12695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:09.693892 12695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:09.693921 12695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:09.695065 12695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:09.695098 12695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:09.695122 12695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:09.696480 12695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:09.696656 12695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:09.696841 12695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:09.701581 12703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:09.701571 12701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:09.701572 12700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:09.701714 12695 server_base.cc:1034] running on GCE node
I20260619 10:26:09.701990 12695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:09.702203 12695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:09.703341 12695 hybrid_clock.cc:648] HybridClock initialized: now 1781864769703331 us; error 23 us; skew 500 ppm
Jun 19 10:26:09 dist-test-slave-hw7q krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864769, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:09.705638 12695 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:26:09.706058 12695 webserver.cc:460] Webserver started at http://127.9.90.65:41869/ using document root <none> and password file <none>
I20260619 10:26:09.706259 12695 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:09.706310 12695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:09.706419 12695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:09.707387 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "f85347a6fc7c4e5cb250db726cf58531"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "7b183bd16bfbce4a15da5d77da5984f1"
server_key_iv: "d2f66e2fa59d5cc5a07d655dcbfbd122"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.707687 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "f85347a6fc7c4e5cb250db726cf58531"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "7b183bd16bfbce4a15da5d77da5984f1"
server_key_iv: "d2f66e2fa59d5cc5a07d655dcbfbd122"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.708824 12695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:09.709419 12710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:09.709559 12695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:09.709638 12695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "f85347a6fc7c4e5cb250db726cf58531"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "7b183bd16bfbce4a15da5d77da5984f1"
server_key_iv: "d2f66e2fa59d5cc5a07d655dcbfbd122"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.709689 12695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:09.719715 12695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:09.721277 12695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:09.721391 12695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:09.721580 12695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:09.721896 12695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:09.721931 12695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:09.721951 12695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:09.721973 12695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:09.727099 12695 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:39249
I20260619 10:26:09.727159 12823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:39249 every 8 connection(s)
I20260619 10:26:09.727444 12695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:09.730046 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 12695
I20260619 10:26:09.730115 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:09.730288 9577 external_mini_cluster.cc:1428] Setting key 513211fb41d1e4603ff0775df073aedb
Jun 19 10:26:09 dist-test-slave-hw7q krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864769, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:09.736016 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:09.728070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:46867 (local address 127.9.90.126:42571)
0619 10:26:09.728139 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:09.728141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:09.728285 (+ 144us) server_negotiation.cc:407] Connection header received
0619 10:26:09.728725 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:09.728727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:09.728739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:09.728800 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:09.729603 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.730195 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.731318 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.731531 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.731584 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:09.733493 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:09.733494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:09.733494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:09.733501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:09.734827 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:09.735178 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:09.735179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:09.735179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:09.735236 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:09.735449 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:09.735450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:09.735451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:09.735555 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:09.735607 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:09.735852 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 10:26:09.735930 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:26:09.736160 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:09.728038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42571 (local address 127.9.90.65:46867)
0619 10:26:09.728179 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:09.728211 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:09.728654 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:09.728844 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:09.728847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:09.729363 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:09.729538 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:09.729543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.730274 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.730275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:09.731243 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:09.731246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.731600 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.731601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:09.731730 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:09.732156 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:09.732165 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:09.733413 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:09.734901 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:09.734907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:09.734909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:09.735113 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:09.735297 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:09.735298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:09.735298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:09.735396 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:09.735608 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:09.735614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:09.735738 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 10:26:09.735824 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 10:26:09.735962 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260619 10:26:09.736600 12824 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42571
I20260619 10:26:09.736698 12824 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:09.736878 12824 heartbeater.cc:511] Master 127.9.90.126:42571 requested a full tablet report, sending...
I20260619 10:26:09.737378 12633 ts_manager.cc:194] Registered new tserver with Master: f85347a6fc7c4e5cb250db726cf58531 (127.9.90.65:39249)
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
I20260619 10:26:09.738019 12633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:46867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:26:09.751763 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42571
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:09.829130 12829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:09.829329 12829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:09.829365 12829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:09.830493 12829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:09.830530 12829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:09.830564 12829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:09.831907 12829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:09.832123 12829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:09.832310 12829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:09.836647 12834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:09.836658 12835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:09.836795 12837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:09.836927 12829 server_base.cc:1034] running on GCE node
I20260619 10:26:09.837194 12829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:09.837401 12829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:09.838492 12829 hybrid_clock.cc:648] HybridClock initialized: now 1781864769838470 us; error 34 us; skew 500 ppm
Jun 19 10:26:09 dist-test-slave-hw7q krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864769, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:09.840837 12829 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:26:09.841256 12829 webserver.cc:460] Webserver started at http://127.9.90.66:34835/ using document root <none> and password file <none>
I20260619 10:26:09.841452 12829 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:09.841497 12829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:09.841614 12829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:09.842602 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "ffabd6fdf8b34fa9a812432232391dcb"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "c7811d3ca08ac5d7993873455d6cd85f"
server_key_iv: "ca6c4026efd5500a510222a8a2383a09"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.842947 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "ffabd6fdf8b34fa9a812432232391dcb"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "c7811d3ca08ac5d7993873455d6cd85f"
server_key_iv: "ca6c4026efd5500a510222a8a2383a09"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.844111 12829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:09.844699 12844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:09.844843 12829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:09.844918 12829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "ffabd6fdf8b34fa9a812432232391dcb"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "c7811d3ca08ac5d7993873455d6cd85f"
server_key_iv: "ca6c4026efd5500a510222a8a2383a09"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.844980 12829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:09.856967 12829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:09.858904 12829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:09.859032 12829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:09.859215 12829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:09.859476 12829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:09.859510 12829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:09.859535 12829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:09.859552 12829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:09.864909 12829 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:46379
I20260619 10:26:09.864969 12957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:46379 every 8 connection(s)
I20260619 10:26:09.865262 12829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:09.865603 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 12829
I20260619 10:26:09.865700 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:09.865916 9577 external_mini_cluster.cc:1428] Setting key edab37168aa0effdb312596f7746f275
Jun 19 10:26:09 dist-test-slave-hw7q krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864769, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
I20260619 10:26:09.874343 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:09.865988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:42637 (local address 127.9.90.126:42571)
0619 10:26:09.866035 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:09.866037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:09.866247 (+ 210us) server_negotiation.cc:407] Connection header received
0619 10:26:09.866828 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:09.866829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:09.866841 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:09.866904 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:09.867809 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.868290 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.869312 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.869612 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.869685 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:09.871617 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:09.871618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:09.871619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:09.871626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:09.873078 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:09.873479 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:09.873480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:09.873480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:09.873541 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:09.873727 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:09.873728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:09.873728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:09.873863 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:09.873908 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:09.874158 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 10:26:09.874239 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:09.874459 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:09.865980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42571 (local address 127.9.90.66:42637)
0619 10:26:09.866120 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:09.866156 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:09.866735 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:09.866955 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:09.866958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:09.867471 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:09.867715 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:09.867721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.868371 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.868373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:09.869227 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:09.869229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:09.869705 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:09.869706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:09.869829 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:09.870276 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:09.870283 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:09.871534 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:09.873164 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:09.873169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:09.873170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:09.873409 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:09.873604 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:09.873605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:09.873605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:09.873691 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:09.873909 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:09.873912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:09.874032 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 10:26:09.874122 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 10:26:09.874261 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260619 10:26:09.874897 12958 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42571
I20260619 10:26:09.874972 12958 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:09.875136 12958 heartbeater.cc:511] Master 127.9.90.126:42571 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
I20260619 10:26:09.875495 12633 ts_manager.cc:194] Registered new tserver with Master: ffabd6fdf8b34fa9a812432232391dcb (127.9.90.66:46379)
I20260619 10:26:09.875851 12633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:42637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:26:09.887486 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42571
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:09.963903 12963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:09.964102 12963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:09.964123 12963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:09.965418 12963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:09.965474 12963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:09.965509 12963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:09.966990 12963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:09.967226 12963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:09.967448 12963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:09.972126 12968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:09.972155 12969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:09.972134 12971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:09.972152 12963 server_base.cc:1034] running on GCE node
I20260619 10:26:09.972572 12963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:09.972806 12963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:09.973929 12963 hybrid_clock.cc:648] HybridClock initialized: now 1781864769973920 us; error 27 us; skew 500 ppm
Jun 19 10:26:09 dist-test-slave-hw7q krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864769, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:09.976173 12963 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:26:09.976612 12963 webserver.cc:460] Webserver started at http://127.9.90.67:34503/ using document root <none> and password file <none>
I20260619 10:26:09.976823 12963 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:09.976871 12963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:09.976992 12963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:09.978047 12963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "4d7af4bcf6c546bb83c917cde7c058ed"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "c0182a72fa5dafe59a9e42d76e756605"
server_key_iv: "0b001219dee9752de6366e15da6dde6c"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.978367 12963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "4d7af4bcf6c546bb83c917cde7c058ed"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "c0182a72fa5dafe59a9e42d76e756605"
server_key_iv: "0b001219dee9752de6366e15da6dde6c"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.979481 12963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:09.980075 12978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:09.980218 12963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:09.980314 12963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "4d7af4bcf6c546bb83c917cde7c058ed"
format_stamp: "Formatted at 2026-06-19 10:26:09 on dist-test-slave-hw7q"
server_key: "c0182a72fa5dafe59a9e42d76e756605"
server_key_iv: "0b001219dee9752de6366e15da6dde6c"
server_key_version: "encryptionkey@0"
I20260619 10:26:09.980370 12963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:09.994962 12963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:09.996498 12963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:09.996616 12963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:09.996793 12963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:09.997051 12963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:09.997083 12963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:09.997109 12963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:09.997133 12963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:10.002315 12963 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:32835
I20260619 10:26:10.002372 13091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:32835 every 8 connection(s)
I20260619 10:26:10.002633 12963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:26:10 dist-test-slave-hw7q krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864769, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:10.011397 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 12963
I20260619 10:26:10.011497 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:10.011521 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:10.003253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:53825 (local address 127.9.90.126:42571)
0619 10:26:10.003325 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:10.003327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:10.003461 (+ 134us) server_negotiation.cc:407] Connection header received
0619 10:26:10.003912 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:10.003913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:10.003922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:10.003969 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:10.004934 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:10.005417 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:10.006467 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:10.006676 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:10.006744 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:10.008960 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:10.008962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:10.008962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:10.008969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:10.010239 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:10.010606 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:10.010607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:10.010607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:10.010660 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:10.010883 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:10.010884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:10.010884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:10.010998 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:10.011053 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:10.011336 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 10:26:10.011429 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 10:26:10.011659 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:10.003221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42571 (local address 127.9.90.67:53825)
0619 10:26:10.003359 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:10.003391 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:10.003838 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:10.004027 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:10.004030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:10.004564 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:10.004842 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:10.004849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:10.005479 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:10.005480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:10.006355 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:10.006358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:10.006777 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:10.006778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:10.006886 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:10.007422 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:10.007430 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:10.008857 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:10.010314 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:10.010321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:10.010323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:10.010539 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:10.010717 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:10.010718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:10.010718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:10.010826 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:10.011053 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:10.011059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:10.011193 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 10:26:10.011280 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 10:26:10.011440 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260619 10:26:10.011701 9577 external_mini_cluster.cc:1428] Setting key ea320058d07785cfb0b468fd445f4c2f
I20260619 10:26:10.012178 13092 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42571
I20260619 10:26:10.012284 13092 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:10.012460 13092 heartbeater.cc:511] Master 127.9.90.126:42571 requested a full tablet report, sending...
I20260619 10:26:10.012810 12632 ts_manager.cc:194] Registered new tserver with Master: 4d7af4bcf6c546bb83c917cde7c058ed (127.9.90.67:32835)
I20260619 10:26:10.013192 12632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:53825
I20260619 10:26:10.023692 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 10:26:10 dist-test-slave-hw7q krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864770, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 10:26:10 dist-test-slave-hw7q krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864770, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:10.044996 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:10.038534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.9.90.126:42571)
0619 10:26:10.038604 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:10.038606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:10.038674 (+ 68us) server_negotiation.cc:407] Connection header received
0619 10:26:10.038747 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:10.038748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:10.038759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:10.038813 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:10.039384 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:10.039872 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:10.040588 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:10.040782 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:10.040836 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:10.042938 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:10.042940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:10.042940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:10.042949 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:10.043994 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:10.044276 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:10.044277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:10.044277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:10.044332 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:10.044494 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:10.044494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:10.044495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:10.044637 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:10.044723 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:10.044828 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 10:26:10.044902 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 10:26:10.045230 12632 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44080
I20260619 10:26:10.045728 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 12695
I20260619 10:26:10.049688 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 12829
I20260619 10:26:10.053907 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 12963
I20260619 10:26:10.057446 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 12601
2026-06-19T10:26:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1661 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:10 dist-test-slave-hw7q krb5kdc[13104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:10 dist-test-slave-hw7q krb5kdc[13104](info): set up 2 sockets
Jun 19 10:26:10 dist-test-slave-hw7q krb5kdc[13104](info): commencing operation
krb5kdc: starting...
W20260619 10:26:11.005648 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:11.115689 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:39813
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:37945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:39813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:11.193252 13117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:11.193475 13117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:11.193500 13117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:11.194659 13117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:11.194701 13117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:11.194715 13117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:11.194725 13117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:11.194736 13117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:11.196161 13117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:37945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:39813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:39813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:11.196337 13117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:11.196496 13117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:11.200860 13123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:11.200873 13122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:11.200867 13125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:11.201146 13117 server_base.cc:1034] running on GCE node
I20260619 10:26:11.201381 13117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:11.201647 13117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:11.202800 13117 hybrid_clock.cc:648] HybridClock initialized: now 1781864771202785 us; error 31 us; skew 500 ppm
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:11.209607 13117 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:11.210047 13117 webserver.cc:460] Webserver started at http://127.9.90.126:35995/ using document root <none> and password file <none>
I20260619 10:26:11.210253 13117 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:11.210305 13117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:11.210427 13117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:11.211395 13117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "2372ebcd72714b32974ffec131cb81a2"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "435a7c93c22ecdd4890a60588a633b47"
server_key_iv: "f40dbe57f7f464d8a10f852ead712b10"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.211691 13117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "2372ebcd72714b32974ffec131cb81a2"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "435a7c93c22ecdd4890a60588a633b47"
server_key_iv: "f40dbe57f7f464d8a10f852ead712b10"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.212849 13117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:11.213446 13132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.213618 13117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:11.213696 13117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "2372ebcd72714b32974ffec131cb81a2"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "435a7c93c22ecdd4890a60588a633b47"
server_key_iv: "f40dbe57f7f464d8a10f852ead712b10"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.213754 13117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:11.229648 13117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:11.231606 13117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:11.231732 13117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:11.235291 13117 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:39813
I20260619 10:26:11.235325 13184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:39813 every 8 connection(s)
I20260619 10:26:11.235733 13117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:11.236334 13185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:11.238576 13185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2: Bootstrap starting.
I20260619 10:26:11.239113 13185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:11.239377 13185 log.cc:824] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:11.239905 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 13117
I20260619 10:26:11.239998 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:11.240082 13185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2: No bootstrap required, opened a new log
I20260619 10:26:11.240185 9577 external_mini_cluster.cc:1428] Setting key 697056b9e804e7fea3204a72a049116d
I20260619 10:26:11.242651 13185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2372ebcd72714b32974ffec131cb81a2" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39813 } }
I20260619 10:26:11.242786 13185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:11.242808 13185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2372ebcd72714b32974ffec131cb81a2, State: Initialized, Role: FOLLOWER
I20260619 10:26:11.242913 13185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2372ebcd72714b32974ffec131cb81a2" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39813 } }
I20260619 10:26:11.242978 13185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:11.243001 13185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:11.243045 13185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:11.243630 13185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2372ebcd72714b32974ffec131cb81a2" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39813 } }
I20260619 10:26:11.243731 13185 leader_election.cc:302] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2372ebcd72714b32974ffec131cb81a2; no voters:
I20260619 10:26:11.243878 13185 leader_election.cc:288] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:11.243918 13190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:11.244046 13190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [term 1 LEADER]: Becoming Leader. State: Replica: 2372ebcd72714b32974ffec131cb81a2, State: Running, Role: LEADER
I20260619 10:26:11.244097 13185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:11.244159 13190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2372ebcd72714b32974ffec131cb81a2" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39813 } }
I20260619 10:26:11.244439 13190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2372ebcd72714b32974ffec131cb81a2. Latest consensus state: current_term: 1 leader_uuid: "2372ebcd72714b32974ffec131cb81a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2372ebcd72714b32974ffec131cb81a2" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39813 } } }
I20260619 10:26:11.244510 13190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:11.244451 13192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2372ebcd72714b32974ffec131cb81a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2372ebcd72714b32974ffec131cb81a2" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39813 } } }
I20260619 10:26:11.244752 13192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:11.245383 13195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:11.245857 13195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:11.247838 13195 catalog_manager.cc:1347] Generated new cluster ID: 9bc24fc8251b4ece8275fa721ff93510
I20260619 10:26:11.247887 13195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:11.250497 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:11.240631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.9.90.126:39813)
0619 10:26:11.240790 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:11.240800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:11.240824 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:26:11.241294 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:11.241300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:11.241344 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:11.241449 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:11.242214 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.243204 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.244212 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.244944 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.245043 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:11.247745 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:11.247748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:11.247748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:11.247756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:11.249231 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.249529 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.249531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.249532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.249613 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.249777 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.249778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.249779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.250006 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:11.250159 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:11.250231 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 10:26:11.250349 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260619 10:26:11.255915 13195 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:11.256381 13195 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:11.261373 13195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2372ebcd72714b32974ffec131cb81a2: Generated new TSK 0
I20260619 10:26:11.261521 13195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:26:11.289173 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39813
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:37945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:11.388111 13211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:11.388301 13211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:11.388322 13211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:11.389467 13211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:11.389505 13211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:11.389530 13211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:11.391002 13211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:37945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:11.391213 13211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:11.391410 13211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:11.395999 13217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:11.396082 13216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:11.396191 13211 server_base.cc:1034] running on GCE node
W20260619 10:26:11.396005 13219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:11.396466 13211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:11.396687 13211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:11.397833 13211 hybrid_clock.cc:648] HybridClock initialized: now 1781864771397820 us; error 36 us; skew 500 ppm
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:11.400270 13211 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:26:11.400672 13211 webserver.cc:460] Webserver started at http://127.9.90.65:43699/ using document root <none> and password file <none>
I20260619 10:26:11.400858 13211 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:11.400897 13211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:11.401014 13211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:11.402041 13211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "22a4bb139bf546258227af27479ac1f5"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "8e662e0838ed2c6010bb227538a84d0a"
server_key_iv: "90b44e8f48b0c641c3479b3be524f419"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.402356 13211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "22a4bb139bf546258227af27479ac1f5"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "8e662e0838ed2c6010bb227538a84d0a"
server_key_iv: "90b44e8f48b0c641c3479b3be524f419"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.403501 13211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:11.404135 13226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.404292 13211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:11.404368 13211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "22a4bb139bf546258227af27479ac1f5"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "8e662e0838ed2c6010bb227538a84d0a"
server_key_iv: "90b44e8f48b0c641c3479b3be524f419"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.404417 13211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:11.420054 13211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:11.421739 13211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:11.421903 13211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:11.422097 13211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:11.422345 13211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:11.422374 13211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.422395 13211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:11.422406 13211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.427517 13211 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:34171
I20260619 10:26:11.427588 13339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:34171 every 8 connection(s)
I20260619 10:26:11.427865 13211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:11.433706 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 13211
I20260619 10:26:11.433817 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:11.434016 9577 external_mini_cluster.cc:1428] Setting key a44c042212c7064a3a91085f12826720
I20260619 10:26:11.436518 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:11.428531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:56473 (local address 127.9.90.126:39813)
0619 10:26:11.428603 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:11.428606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:11.428756 (+ 150us) server_negotiation.cc:407] Connection header received
0619 10:26:11.429628 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:11.429629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:11.429638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:11.429687 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:11.430258 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.430754 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.431657 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.431851 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.431894 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:11.433754 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:11.433756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:11.433756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:11.433773 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:11.435261 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.435633 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.435634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.435634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.435695 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.435912 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.435914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.435914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.436023 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:11.436081 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:11.436348 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 10:26:11.436431 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 10:26:11.436658 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:11.428502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:39813 (local address 127.9.90.65:56473)
0619 10:26:11.428646 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:11.428680 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:11.429541 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:11.429741 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:11.429744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:11.430003 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:11.430188 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:11.430192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.430826 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.430827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:11.431600 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:11.431602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.431905 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.431906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:11.431998 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:11.432410 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:11.432418 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:11.433665 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:11.435366 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:11.435371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:11.435373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:11.435573 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:11.435749 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:11.435749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:11.435750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:11.435851 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:11.436081 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:11.436087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:11.436213 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 10:26:11.436318 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 10:26:11.436463 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260619 10:26:11.437176 13340 heartbeater.cc:347] Connected to a master server at 127.9.90.126:39813
I20260619 10:26:11.437271 13340 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:11.437440 13340 heartbeater.cc:511] Master 127.9.90.126:39813 requested a full tablet report, sending...
I20260619 10:26:11.437970 13149 ts_manager.cc:194] Registered new tserver with Master: 22a4bb139bf546258227af27479ac1f5 (127.9.90.65:34171)
I20260619 10:26:11.438537 13149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:56473
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:26:11.455767 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39813
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:37945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:11.533679 13345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:11.533914 13345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:11.533941 13345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:11.535079 13345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:11.535120 13345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:11.535145 13345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:11.536665 13345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:37945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:11.536881 13345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:11.537079 13345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:11.541641 13353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:11.541683 13350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:11.541911 13345 server_base.cc:1034] running on GCE node
W20260619 10:26:11.542030 13351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:11.542294 13345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:11.542506 13345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:11.543649 13345 hybrid_clock.cc:648] HybridClock initialized: now 1781864771543634 us; error 31 us; skew 500 ppm
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:11.546221 13345 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:26:11.546598 13345 webserver.cc:460] Webserver started at http://127.9.90.66:39907/ using document root <none> and password file <none>
I20260619 10:26:11.546770 13345 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:11.546809 13345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:11.546898 13345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:11.547860 13345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "e2d1ee6630ad4b0cba049bd9efdaf5da"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "3cb6c271bc7dbd71802f2147351c99c0"
server_key_iv: "ecc5843fe1cae2a04c221f0a5b0e1d09"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.548137 13345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "e2d1ee6630ad4b0cba049bd9efdaf5da"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "3cb6c271bc7dbd71802f2147351c99c0"
server_key_iv: "ecc5843fe1cae2a04c221f0a5b0e1d09"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.549213 13345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:26:11.549904 13360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.550055 13345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.550134 13345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "e2d1ee6630ad4b0cba049bd9efdaf5da"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "3cb6c271bc7dbd71802f2147351c99c0"
server_key_iv: "ecc5843fe1cae2a04c221f0a5b0e1d09"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.550201 13345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:11.565557 13345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:11.567616 13345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:11.567747 13345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:11.567945 13345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:11.568241 13345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:11.568287 13345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.568315 13345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:11.568337 13345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.573643 13345 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:43601
I20260619 10:26:11.573702 13473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:43601 every 8 connection(s)
I20260619 10:26:11.573997 13345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:11.580056 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 13345
I20260619 10:26:11.580143 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:11.580353 9577 external_mini_cluster.cc:1428] Setting key 169ce85b9657975baa050b6d1f36b3ea
I20260619 10:26:11.582548 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:11.574634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:48911 (local address 127.9.90.126:39813)
0619 10:26:11.574687 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:11.574689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:11.574904 (+ 215us) server_negotiation.cc:407] Connection header received
0619 10:26:11.575813 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:11.575814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:11.575823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:11.575858 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:11.576413 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.576910 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.577883 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.578065 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.578114 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:11.579977 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:11.579979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:11.579979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:11.579986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:11.581310 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.581645 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.581646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.581646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.581699 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.581940 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.581941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.581941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.582072 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:11.582135 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:11.582374 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 10:26:11.582460 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:26:11.582683 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:11.574614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:39813 (local address 127.9.90.66:48911)
0619 10:26:11.574765 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:11.574824 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:11.575695 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:11.575920 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:11.575924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:11.576162 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:11.576350 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:11.576354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.576971 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.576971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:11.577745 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:11.577747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.578124 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.578124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:11.578226 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:11.578621 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:11.578629 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:11.579886 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:11.581376 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:11.581382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:11.581383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:11.581584 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:11.581752 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:11.581752 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:11.581753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:11.581887 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:11.582126 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:11.582128 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:11.582234 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 10:26:11.582333 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 10:26:11.582482 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260619 10:26:11.583060 13474 heartbeater.cc:347] Connected to a master server at 127.9.90.126:39813
I20260619 10:26:11.583138 13474 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:11.583302 13474 heartbeater.cc:511] Master 127.9.90.126:39813 requested a full tablet report, sending...
I20260619 10:26:11.583640 13149 ts_manager.cc:194] Registered new tserver with Master: e2d1ee6630ad4b0cba049bd9efdaf5da (127.9.90.66:43601)
I20260619 10:26:11.584000 13149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:48911
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:26:11.601435 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39813
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:37945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:11.679220 13479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:11.679427 13479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:11.679461 13479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:11.680598 13479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:11.680637 13479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:11.680671 13479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:11.682230 13479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:37945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:11.682471 13479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:11.682669 13479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:11.687428 13487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:11.687422 13485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:11.687423 13484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:11.687592 13479 server_base.cc:1034] running on GCE node
I20260619 10:26:11.687826 13479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:11.688033 13479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:11.689162 13479 hybrid_clock.cc:648] HybridClock initialized: now 1781864771689146 us; error 29 us; skew 500 ppm
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:11.691501 13479 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:26:11.691919 13479 webserver.cc:460] Webserver started at http://127.9.90.67:40603/ using document root <none> and password file <none>
I20260619 10:26:11.692116 13479 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:11.692160 13479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:11.692293 13479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:11.693295 13479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "b3b2a41dced84f0ab76c955488a30ab2"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "9ac6598e30a34b4267d8deee4700e398"
server_key_iv: "184abe4cff9182cfaf6667f1cc5eb89b"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.693614 13479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "b3b2a41dced84f0ab76c955488a30ab2"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "9ac6598e30a34b4267d8deee4700e398"
server_key_iv: "184abe4cff9182cfaf6667f1cc5eb89b"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.694840 13479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:11.695497 13494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.695647 13479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:11.695726 13479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "b3b2a41dced84f0ab76c955488a30ab2"
format_stamp: "Formatted at 2026-06-19 10:26:11 on dist-test-slave-hw7q"
server_key: "9ac6598e30a34b4267d8deee4700e398"
server_key_iv: "184abe4cff9182cfaf6667f1cc5eb89b"
server_key_version: "encryptionkey@0"
I20260619 10:26:11.695791 13479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:11.710376 13479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:11.712405 13479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:11.712535 13479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:11.712733 13479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:11.713008 13479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:11.713039 13479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.713066 13479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:11.713090 13479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:11.718789 13479 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:34727
I20260619 10:26:11.718859 13607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:34727 every 8 connection(s)
I20260619 10:26:11.719138 13479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:11.725436 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 13479
I20260619 10:26:11.725536 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:11.725953 9577 external_mini_cluster.cc:1428] Setting key b0ec73a41a8961684df2f4c46d2ac9b2
I20260619 10:26:11.727846 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:11.719779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:47359 (local address 127.9.90.126:39813)
0619 10:26:11.719833 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:11.719835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:11.720038 (+ 203us) server_negotiation.cc:407] Connection header received
0619 10:26:11.720915 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:11.720915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:11.720924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:11.720979 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:11.721654 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.722248 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.723189 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.723395 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.723449 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:11.725389 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:11.725391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:11.725391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:11.725400 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:11.726634 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.727002 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.727003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.727004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.727055 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.727268 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.727269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.727270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.727373 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:11.727427 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:11.727675 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 10:26:11.727750 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:26:11.728006 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:11.719750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:39813 (local address 127.9.90.67:47359)
0619 10:26:11.719918 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:11.719959 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:11.720833 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:11.721045 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:11.721048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:11.721320 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:11.721507 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:11.721512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.722332 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.722333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:11.723118 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:11.723120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.723455 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.723456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:11.723548 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:11.723969 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:11.723977 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:11.725301 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:11.726710 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:11.726716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:11.726718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:11.726932 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:11.727108 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:11.727109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:11.727109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:11.727213 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:11.727427 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:11.727433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:11.727563 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 10:26:11.727649 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 10:26:11.727799 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260619 10:26:11.728448 13608 heartbeater.cc:347] Connected to a master server at 127.9.90.126:39813
I20260619 10:26:11.728523 13608 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:11.728682 13608 heartbeater.cc:511] Master 127.9.90.126:39813 requested a full tablet report, sending...
I20260619 10:26:11.728991 13149 ts_manager.cc:194] Registered new tserver with Master: b3b2a41dced84f0ab76c955488a30ab2 (127.9.90.67:34727)
I20260619 10:26:11.729351 13149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:47359
I20260619 10:26:11.737401 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:26:11.757838 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:11.751110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.9.90.65:34171)
0619 10:26:11.751247 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:11.751251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:11.751267 (+ 16us) server_negotiation.cc:407] Connection header received
0619 10:26:11.751291 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:11.751293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:11.751306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:11.751371 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:11.751824 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.752525 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.753230 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.753546 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.753599 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:11.755564 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:11.755572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:11.755572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:11.755578 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:11.756947 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.757227 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.757229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.757229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.757295 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.757418 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.757418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.757418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.757532 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:11.757632 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:11.757668 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 10:26:11.757723 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:11.764806 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:11.758998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.9.90.126:39813)
0619 10:26:11.759048 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:11.759050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:11.759061 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:26:11.759114 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:11.759115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:11.759123 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:11.759165 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:11.759625 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.760108 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.760694 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:11.760870 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:11.760917 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:11.762960 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:11.762962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:11.762962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:11.762969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:11.763985 (+ 1016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.764247 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.764248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.764248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.764293 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:11.764413 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:11.764414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:11.764414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:11.764533 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:11.764599 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:11.764671 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 10:26:11.764726 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 10:26:11.765026 13149 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40314
I20260619 10:26:11.765355 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 13211
I20260619 10:26:11.770100 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 13345
I20260619 10:26:11.773589 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 13479
I20260619 10:26:11.776786 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 13117
2026-06-19T10:26:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1718 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13620](info): set up 2 sockets
Jun 19 10:26:11 dist-test-slave-hw7q krb5kdc[13620](info): commencing operation
krb5kdc: starting...
W20260619 10:26:12.725509 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:12 dist-test-slave-hw7q krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:12.831915 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45693
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:45693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:12.909477 13633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:12.909667 13633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:12.909690 13633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:12.910851 13633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:12.910897 13633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:12.910909 13633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:12.910920 13633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:12.910931 13633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:12.912391 13633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:45693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:12.912611 13633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:12.912817 13633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:12.917372 13639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:12.917384 13638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:12.917580 13641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:12.918020 13633 server_base.cc:1034] running on GCE node
I20260619 10:26:12.918244 13633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:12.918512 13633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:12.919638 13633 hybrid_clock.cc:648] HybridClock initialized: now 1781864772919627 us; error 33 us; skew 500 ppm
Jun 19 10:26:12 dist-test-slave-hw7q krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864772, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:12.921924 13633 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:12.922248 13633 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:12.922302 13633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:12.922392 13633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:12.923491 13633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "b1f9482072704d90b062e298dad59fda"
format_stamp: "Formatted at 2026-06-19 10:26:12 on dist-test-slave-hw7q"
server_key: "956a565c116ad4ce88b982d6e6b90f95"
server_key_iv: "0951be6ef249a42829afc3b7bfe8ed5b"
server_key_version: "encryptionkey@0"
I20260619 10:26:12.923774 13633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "b1f9482072704d90b062e298dad59fda"
format_stamp: "Formatted at 2026-06-19 10:26:12 on dist-test-slave-hw7q"
server_key: "956a565c116ad4ce88b982d6e6b90f95"
server_key_iv: "0951be6ef249a42829afc3b7bfe8ed5b"
server_key_version: "encryptionkey@0"
I20260619 10:26:12.924932 13633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:26:12.925721 13647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:12.925937 13633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:26:12.926023 13633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "b1f9482072704d90b062e298dad59fda"
format_stamp: "Formatted at 2026-06-19 10:26:12 on dist-test-slave-hw7q"
server_key: "956a565c116ad4ce88b982d6e6b90f95"
server_key_iv: "0951be6ef249a42829afc3b7bfe8ed5b"
server_key_version: "encryptionkey@0"
I20260619 10:26:12.926088 13633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:12.945904 13633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:12.948127 13633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:12.948279 13633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:12.952353 13633 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:45693
I20260619 10:26:12.952385 13699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:45693 every 8 connection(s)
I20260619 10:26:12.952682 13633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:12.953581 13700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:12.955840 13700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda: Bootstrap starting.
I20260619 10:26:12.956235 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 13633
I20260619 10:26:12.956326 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:12.956434 13700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:12.956494 9577 external_mini_cluster.cc:1428] Setting key bf407c763b40fee4a293a8fccc9325bf
I20260619 10:26:12.956718 13700 log.cc:824] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:12.957458 13700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda: No bootstrap required, opened a new log
I20260619 10:26:12.959492 13700 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9482072704d90b062e298dad59fda" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45693 } }
I20260619 10:26:12.959669 13700 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:12.959703 13700 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f9482072704d90b062e298dad59fda, State: Initialized, Role: FOLLOWER
I20260619 10:26:12.959830 13700 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9482072704d90b062e298dad59fda" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45693 } }
I20260619 10:26:12.959905 13700 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:12.959939 13700 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:12.959980 13700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:12.960567 13700 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9482072704d90b062e298dad59fda" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45693 } }
I20260619 10:26:12.960659 13700 leader_election.cc:302] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f9482072704d90b062e298dad59fda; no voters:
I20260619 10:26:12.960801 13700 leader_election.cc:288] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:12.960839 13705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:12.960984 13705 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [term 1 LEADER]: Becoming Leader. State: Replica: b1f9482072704d90b062e298dad59fda, State: Running, Role: LEADER
I20260619 10:26:12.961054 13700 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:12.961164 13705 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9482072704d90b062e298dad59fda" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45693 } }
I20260619 10:26:12.961508 13706 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f9482072704d90b062e298dad59fda. Latest consensus state: current_term: 1 leader_uuid: "b1f9482072704d90b062e298dad59fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9482072704d90b062e298dad59fda" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45693 } } }
I20260619 10:26:12.961568 13706 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:12.961812 13712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:12.962445 13712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 10:26:12 dist-test-slave-hw7q krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:12.962821 13707 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f9482072704d90b062e298dad59fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f9482072704d90b062e298dad59fda" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45693 } } }
I20260619 10:26:12.963575 13707 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:12.964826 13712 catalog_manager.cc:1347] Generated new cluster ID: 117868be742045239b2e21c5b4f6adef
I20260619 10:26:12.964876 13712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:12.970767 13712 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:12.971262 13712 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:12.972432 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:12.956980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59714 (local address 127.9.90.126:45693)
0619 10:26:12.957132 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:12.957140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:12.957164 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:26:12.957595 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:12.957603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:12.957653 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:12.957783 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:12.958527 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:12.959372 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:12.960384 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:12.960846 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:12.961232 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:12.963921 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:12.963925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:12.963926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:12.963938 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:12.970785 (+ 6847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:12.971209 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:12.971212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:12.971212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:12.971311 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:12.971503 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:12.971504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:12.971504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:12.971770 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:12.971931 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:12.972047 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 10:26:12.972193 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260619 10:26:12.975818 13712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda: Generated new TSK 0
I20260619 10:26:12.975979 13712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:26:13.002277 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45693
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:13.084249 13726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:13.084445 13726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:13.084477 13726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:13.085599 13726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:13.085637 13726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:13.085672 13726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:13.087169 13726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:13.087386 13726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:13.087584 13726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:13.092145 13731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.092085 13734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:13.092140 13726 server_base.cc:1034] running on GCE node
W20260619 10:26:13.092092 13732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:13.092543 13726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:13.092785 13726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:13.093917 13726 hybrid_clock.cc:648] HybridClock initialized: now 1781864773093904 us; error 28 us; skew 500 ppm
Jun 19 10:26:13 dist-test-slave-hw7q krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864773, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:13.096231 13726 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:26:13.096554 13726 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:13.096618 13726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:13.096724 13726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:13.097669 13726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "bd75a010eae04296b1d521538df3f06e"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "5a37205aa0a719839ea2d94c88e0a71d"
server_key_iv: "c75ed2a6a52ba6ee84331ee6d4fa6318"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.098027 13726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "bd75a010eae04296b1d521538df3f06e"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "5a37205aa0a719839ea2d94c88e0a71d"
server_key_iv: "c75ed2a6a52ba6ee84331ee6d4fa6318"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.099119 13726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:13.099740 13740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.099900 13726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:13.099977 13726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "bd75a010eae04296b1d521538df3f06e"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "5a37205aa0a719839ea2d94c88e0a71d"
server_key_iv: "c75ed2a6a52ba6ee84331ee6d4fa6318"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.100040 13726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:13.127182 13726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:13.129307 13726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:13.129467 13726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:13.129683 13726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:13.130055 13726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:13.130092 13726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.130113 13726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:13.130124 13726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.135448 13726 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:43793
I20260619 10:26:13.135490 13853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:43793 every 8 connection(s)
I20260619 10:26:13.135741 13726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:13.137172 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 13726
I20260619 10:26:13.137272 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:13.137434 9577 external_mini_cluster.cc:1428] Setting key 701d0a708a8d33a9b488f366a2ca8d37
Jun 19 10:26:13 dist-test-slave-hw7q krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864773, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
I20260619 10:26:13.145478 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.136362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:45253 (local address 127.9.90.126:45693)
0619 10:26:13.136404 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.136406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.136632 (+ 226us) server_negotiation.cc:407] Connection header received
0619 10:26:13.137128 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.137130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.137143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.137195 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:13.138375 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.138862 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.140212 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.140442 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.140503 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.142880 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:13.142882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:13.142882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:13.142890 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:13.144196 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:13.144565 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:13.144566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:13.144566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:13.144617 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:13.144833 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:13.144834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:13.144834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:13.144956 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:13.145015 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.145284 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.145383 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:26:13.145642 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.136365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:45693 (local address 127.9.90.65:45253)
0619 10:26:13.136518 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.136553 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.137036 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.137288 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.137292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.137603 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:13.138288 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.138295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.138941 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.138942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.140108 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.140111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.140526 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.140527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.140668 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.141148 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:13.141159 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:13.142773 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:13.144279 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:13.144286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:13.144288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:13.144498 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:13.144671 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:13.144671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:13.144672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:13.144780 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:13.145012 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:13.145019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:13.145156 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.145260 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.145416 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260619 10:26:13.146087 13854 heartbeater.cc:347] Connected to a master server at 127.9.90.126:45693
I20260619 10:26:13.146174 13854 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:13.146296 13854 heartbeater.cc:511] Master 127.9.90.126:45693 requested a full tablet report, sending...
I20260619 10:26:13.146739 13664 ts_manager.cc:194] Registered new tserver with Master: bd75a010eae04296b1d521538df3f06e (127.9.90.65:43793)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
I20260619 10:26:13.147481 13664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:45253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:26:13.157956 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45693
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:13.235124 13859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:13.235311 13859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:13.235342 13859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:13.236449 13859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:13.236486 13859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:13.236521 13859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:13.237957 13859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:13.238180 13859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:13.238375 13859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:13.242847 13865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.242894 13864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.242919 13867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:13.243044 13859 server_base.cc:1034] running on GCE node
I20260619 10:26:13.243214 13859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:13.243392 13859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:13.244524 13859 hybrid_clock.cc:648] HybridClock initialized: now 1781864773244512 us; error 24 us; skew 500 ppm
Jun 19 10:26:13 dist-test-slave-hw7q krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864773, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:13.246917 13859 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:26:13.247248 13859 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:13.247313 13859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:13.247447 13859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:13.248481 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "de47a9883146409380d36199af27ca86"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "acdabcbcdf529f8551ce3723ec6c42f8"
server_key_iv: "36af65dc119857fa71f5f331ef029df6"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.248826 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "de47a9883146409380d36199af27ca86"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "acdabcbcdf529f8551ce3723ec6c42f8"
server_key_iv: "36af65dc119857fa71f5f331ef029df6"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.250063 13859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:13.250890 13873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.251044 13859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:13.251134 13859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "de47a9883146409380d36199af27ca86"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "acdabcbcdf529f8551ce3723ec6c42f8"
server_key_iv: "36af65dc119857fa71f5f331ef029df6"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.251207 13859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:13.259608 13859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:13.261734 13859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:13.261905 13859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:13.262095 13859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:13.262439 13859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:13.262475 13859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.262494 13859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:13.262504 13859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.267756 13859 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:34429
I20260619 10:26:13.267812 13986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:34429 every 8 connection(s)
I20260619 10:26:13.268033 13859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:13.271804 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 13859
I20260619 10:26:13.271899 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:13.272091 9577 external_mini_cluster.cc:1428] Setting key 86f09696f578b5af7be41d09c64668d2
Jun 19 10:26:13 dist-test-slave-hw7q krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864773, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:13.277007 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.268651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:38277 (local address 127.9.90.126:45693)
0619 10:26:13.268707 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.268710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.268903 (+ 193us) server_negotiation.cc:407] Connection header received
0619 10:26:13.269386 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.269388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.269401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.269459 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:13.270411 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.270912 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.272043 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.272312 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.272379 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.274534 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:13.274536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:13.274537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:13.274544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:13.275767 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:13.276148 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:13.276150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:13.276150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:13.276204 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:13.276426 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:13.276427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:13.276427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:13.276533 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:13.276588 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.276839 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.276919 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:26:13.277163 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.268651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:45693 (local address 127.9.90.66:38277)
0619 10:26:13.268790 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.268824 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.269286 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.269528 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.269532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.269856 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:13.270321 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.270327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.270992 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.270994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.271962 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.271966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.272390 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.272391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.272525 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.272981 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:13.272992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:13.274449 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:13.275853 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:13.275859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:13.275861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:13.276087 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:13.276257 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:13.276258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:13.276259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:13.276369 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:13.276586 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:13.276593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:13.276723 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.276812 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.276957 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260619 10:26:13.277587 13987 heartbeater.cc:347] Connected to a master server at 127.9.90.126:45693
I20260619 10:26:13.277686 13987 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:13.277860 13987 heartbeater.cc:511] Master 127.9.90.126:45693 requested a full tablet report, sending...
I20260619 10:26:13.278224 13664 ts_manager.cc:194] Registered new tserver with Master: de47a9883146409380d36199af27ca86 (127.9.90.66:34429)
I20260619 10:26:13.278707 13664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:38277
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:26:13.293287 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45693
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:13.369747 13992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:13.369982 13992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:13.370005 13992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:13.371129 13992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:13.371168 13992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:13.371193 13992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:13.372669 13992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:13.372881 13992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:13.373078 13992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:13.377576 14000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:13.377605 13992 server_base.cc:1034] running on GCE node
W20260619 10:26:13.377549 13998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.377544 13997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:13.377940 13992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:13.378147 13992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:13.379283 13992 hybrid_clock.cc:648] HybridClock initialized: now 1781864773379270 us; error 29 us; skew 500 ppm
Jun 19 10:26:13 dist-test-slave-hw7q krb5kdc[13620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864773, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:13.381518 13992 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:26:13.381891 13992 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:13.381943 13992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:13.382067 13992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:13.383019 13992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "845247f61fda4017944ad2b02d61f41f"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "f5d081df9b1681bfc76f42d90210c767"
server_key_iv: "25737d3d24c09d4afec9da3ee6a19206"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.383329 13992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "845247f61fda4017944ad2b02d61f41f"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "f5d081df9b1681bfc76f42d90210c767"
server_key_iv: "25737d3d24c09d4afec9da3ee6a19206"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.384439 13992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:13.385046 14006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.385191 13992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:13.385268 13992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "845247f61fda4017944ad2b02d61f41f"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "f5d081df9b1681bfc76f42d90210c767"
server_key_iv: "25737d3d24c09d4afec9da3ee6a19206"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.385331 13992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:13.392117 13992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:13.393642 13992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:13.393776 13992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:13.393996 13992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:13.394258 13992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:13.394289 13992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.394313 13992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:13.394336 13992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.399574 13992 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:46719
I20260619 10:26:13.399605 14119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:46719 every 8 connection(s)
I20260619 10:26:13.399833 13992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:26:13 dist-test-slave-hw7q krb5kdc[13620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864773, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:13.407073 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 13992
I20260619 10:26:13.407179 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:13.407363 9577 external_mini_cluster.cc:1428] Setting key dffaabf5b13cab95ed4568f3283aed4d
I20260619 10:26:13.407989 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.400460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:54737 (local address 127.9.90.126:45693)
0619 10:26:13.400502 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.400505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.400670 (+ 165us) server_negotiation.cc:407] Connection header received
0619 10:26:13.401097 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.401098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.401108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.401157 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:13.401991 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.402501 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.403506 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.403699 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.403748 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.405525 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:13.405526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:13.405527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:13.405533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:13.406724 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:13.407074 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:13.407074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:13.407075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:13.407117 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:13.407377 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:13.407378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:13.407378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:13.407494 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:13.407549 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.407831 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.407901 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:13.408145 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.400425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:45693 (local address 127.9.90.67:54737)
0619 10:26:13.400559 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.400595 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.401027 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.401207 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.401210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.401438 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:13.401925 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.401931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.402562 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.402563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.403438 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.403441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.403780 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.403781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.403892 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.404262 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:13.404269 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:13.405448 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:13.406787 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:13.406790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:13.406791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:13.407012 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:13.407178 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:13.407179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:13.407180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:13.407318 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:13.407549 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:13.407555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:13.407699 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.407792 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.407944 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260619 10:26:13.408510 14120 heartbeater.cc:347] Connected to a master server at 127.9.90.126:45693
I20260619 10:26:13.408592 14120 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:13.408720 14120 heartbeater.cc:511] Master 127.9.90.126:45693 requested a full tablet report, sending...
I20260619 10:26:13.409016 13664 ts_manager.cc:194] Registered new tserver with Master: 845247f61fda4017944ad2b02d61f41f (127.9.90.67:46719)
I20260619 10:26:13.409349 13664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:54737
I20260619 10:26:13.418866 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:26:13.425076 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.420051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.9.90.126:45693)
0619 10:26:13.420090 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.420092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.420190 (+ 98us) server_negotiation.cc:407] Connection header received
0619 10:26:13.420237 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.420238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.420245 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.420288 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:13.420797 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.421254 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.421960 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.422134 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.422180 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.423135 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:13.423136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:13.423137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:13.423142 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:13.424268 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:13.424516 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:13.424517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:13.424517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:13.424557 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:13.424670 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:13.424670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:13.424671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:13.424805 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:13.424874 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.424946 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.425001 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:26:13.426527 13664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:13.428007 13664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:13.433928 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.429936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:46719 (local address 127.0.0.1:58524)
0619 10:26:13.430059 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.430098 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.430156 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.430569 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.430572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.430574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:13.430636 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.430637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.431935 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.431935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.432997 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.432999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.433609 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.433610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.433689 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.433703 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.433805 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.433858 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":4,"thread_start_us":52,"threads_started":1}
I20260619 10:26:13.433949 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.430057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.9.90.67:46719)
0619 10:26:13.430406 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.430411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.430421 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:13.430459 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.430461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.430477 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.430563 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:13.430717 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.431835 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.433068 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.433531 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.433594 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.433650 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.433828 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.433872 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":71,"threads_started":1}
I20260619 10:26:13.434689 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.430124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:43793 (local address 127.0.0.1:55226)
0619 10:26:13.430659 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.430669 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.430716 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.430898 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.430899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.430901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:13.430941 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.430942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.431736 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.431737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.432996 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.432999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.434453 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.434454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.434517 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.434522 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.434583 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.434621 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260619 10:26:13.434703 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.430222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.9.90.65:43793)
0619 10:26:13.430385 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.430389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.430768 (+ 379us) server_negotiation.cc:407] Connection header received
0619 10:26:13.430810 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.430812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.430827 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.430890 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:13.430997 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.431651 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.433686 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.434369 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.434438 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.434491 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.434598 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.434632 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260619 10:26:13.434655 14054 tablet_service.cc:1467] Processing CreateTablet for tablet d9ca314916a44625bbcdc7611eab6661 (DEFAULT_TABLE table=test-table [id=11d313e3bf3a4aae8d4994c745df4b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:13.435009 14054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ca314916a44625bbcdc7611eab6661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:13.435397 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.429953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:34429 (local address 127.0.0.1:59858)
0619 10:26:13.430244 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.430251 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.430287 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.430662 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.430663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.430665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:13.430702 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.430703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.433089 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.433090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.433752 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.433754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.435183 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.435184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.435250 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.435254 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.435300 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.435336 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":63,"threads_started":1}
I20260619 10:26:13.435456 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.430214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.9.90.66:34429)
0619 10:26:13.430406 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.430411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.430421 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:13.430460 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.430461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.430477 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.430563 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:13.430790 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.431848 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.433882 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.435105 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.435163 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.435217 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.435310 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.435383 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260619 10:26:13.437496 14134 tablet_bootstrap.cc:492] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: Bootstrap starting.
I20260619 10:26:13.438139 14134 tablet_bootstrap.cc:654] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:13.438109 13921 tablet_service.cc:1467] Processing CreateTablet for tablet d9ca314916a44625bbcdc7611eab6661 (DEFAULT_TABLE table=test-table [id=11d313e3bf3a4aae8d4994c745df4b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:13.438184 13788 tablet_service.cc:1467] Processing CreateTablet for tablet d9ca314916a44625bbcdc7611eab6661 (DEFAULT_TABLE table=test-table [id=11d313e3bf3a4aae8d4994c745df4b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:13.438396 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ca314916a44625bbcdc7611eab6661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:13.438515 13788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ca314916a44625bbcdc7611eab6661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:13.438719 14134 log.cc:824] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:13.439482 14134 tablet_bootstrap.cc:492] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: No bootstrap required, opened a new log
I20260619 10:26:13.439553 14134 ts_tablet_manager.cc:1397] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:26:13.440912 14136 tablet_bootstrap.cc:492] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: Bootstrap starting.
I20260619 10:26:13.441393 14134 raft_consensus.cc:348] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } }
I20260619 10:26:13.441547 14134 raft_consensus.cc:374] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:13.441581 14134 raft_consensus.cc:729] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845247f61fda4017944ad2b02d61f41f, State: Initialized, Role: FOLLOWER
I20260619 10:26:13.441577 14137 tablet_bootstrap.cc:492] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e: Bootstrap starting.
I20260619 10:26:13.441672 14136 tablet_bootstrap.cc:654] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:13.441702 14134 consensus_queue.cc:260] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } }
I20260619 10:26:13.442010 14120 heartbeater.cc:503] Master 127.9.90.126:45693 was elected leader, sending a full tablet report...
I20260619 10:26:13.442010 14134 ts_tablet_manager.cc:1428] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 10:26:13.442276 14137 tablet_bootstrap.cc:654] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:13.442299 14136 log.cc:824] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:13.442814 14137 log.cc:824] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:13.443428 14136 tablet_bootstrap.cc:492] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: No bootstrap required, opened a new log
I20260619 10:26:13.443506 14136 ts_tablet_manager.cc:1397] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 10:26:13.444917 14136 raft_consensus.cc:348] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } }
I20260619 10:26:13.445039 14136 raft_consensus.cc:374] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:13.445068 14136 raft_consensus.cc:729] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de47a9883146409380d36199af27ca86, State: Initialized, Role: FOLLOWER
I20260619 10:26:13.445176 14136 consensus_queue.cc:260] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } }
I20260619 10:26:13.445405 14136 ts_tablet_manager.cc:1428] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:26:13.445436 13987 heartbeater.cc:503] Master 127.9.90.126:45693 was elected leader, sending a full tablet report...
I20260619 10:26:13.445752 14137 tablet_bootstrap.cc:492] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e: No bootstrap required, opened a new log
I20260619 10:26:13.445863 14137 ts_tablet_manager.cc:1397] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 10:26:13.447433 14137 raft_consensus.cc:348] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } }
I20260619 10:26:13.447566 14137 raft_consensus.cc:374] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:13.447597 14137 raft_consensus.cc:729] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd75a010eae04296b1d521538df3f06e, State: Initialized, Role: FOLLOWER
I20260619 10:26:13.447703 14137 consensus_queue.cc:260] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } }
I20260619 10:26:13.447935 14137 ts_tablet_manager.cc:1428] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:26:13.447957 13854 heartbeater.cc:503] Master 127.9.90.126:45693 was elected leader, sending a full tablet report...
W20260619 10:26:13.518487 13988 tablet.cc:2369] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:26:13.556146 14138 raft_consensus.cc:482] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:13.556289 14138 raft_consensus.cc:504] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } }
I20260619 10:26:13.556628 14138 leader_election.cc:288] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de47a9883146409380d36199af27ca86 (127.9.90.66:34429), bd75a010eae04296b1d521538df3f06e (127.9.90.65:43793)
I20260619 10:26:13.560256 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.556859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:48413 (local address 127.9.90.66:34429)
0619 10:26:13.556916 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.556918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.556928 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:13.556974 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.556975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.556986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.557038 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:13.557192 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.557990 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.559365 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.559921 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.559981 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.559991 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.560151 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.560185 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:26:13.560256 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.557018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:49185 (local address 127.9.90.65:43793)
0619 10:26:13.557064 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.557066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.557165 (+ 99us) server_negotiation.cc:407] Connection header received
0619 10:26:13.557224 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.557224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.557236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.557300 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:13.557407 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.558087 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.559363 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.559909 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.559981 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.559991 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.560151 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.560185 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:26:13.560276 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.556790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:34429 (local address 127.9.90.67:48413)
0619 10:26:13.556850 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.556862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.556918 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.557051 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.557052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.557055 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:13.557126 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.557128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.558054 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.558055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.559286 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.559287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.559984 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.559985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.560084 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.560089 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.560141 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.560188 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":14}
I20260619 10:26:13.560285 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.556945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:43793 (local address 127.9.90.67:49185)
0619 10:26:13.557099 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.557109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.557164 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.557306 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.557307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.557308 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:13.557352 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.557353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.558143 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.558143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.559285 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:13.559287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.559985 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.559985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:13.560084 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.560089 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.560141 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.560188 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260619 10:26:13.560590 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ca314916a44625bbcdc7611eab6661" candidate_uuid: "845247f61fda4017944ad2b02d61f41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de47a9883146409380d36199af27ca86" is_pre_election: true
I20260619 10:26:13.560726 13941 raft_consensus.cc:2454] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 845247f61fda4017944ad2b02d61f41f in term 0.
I20260619 10:26:13.560891 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ca314916a44625bbcdc7611eab6661" candidate_uuid: "845247f61fda4017944ad2b02d61f41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd75a010eae04296b1d521538df3f06e" is_pre_election: true
I20260619 10:26:13.560984 14010 leader_election.cc:302] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 845247f61fda4017944ad2b02d61f41f, de47a9883146409380d36199af27ca86; no voters:
I20260619 10:26:13.561060 13808 raft_consensus.cc:2454] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 845247f61fda4017944ad2b02d61f41f in term 0.
I20260619 10:26:13.561094 14138 raft_consensus.cc:2781] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:13.561172 14138 raft_consensus.cc:482] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:13.561198 14138 raft_consensus.cc:3037] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:13.561950 14138 raft_consensus.cc:504] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } }
I20260619 10:26:13.562073 14138 leader_election.cc:288] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [CANDIDATE]: Term 1 election: Requested vote from peers de47a9883146409380d36199af27ca86 (127.9.90.66:34429), bd75a010eae04296b1d521538df3f06e (127.9.90.65:43793)
I20260619 10:26:13.562227 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ca314916a44625bbcdc7611eab6661" candidate_uuid: "845247f61fda4017944ad2b02d61f41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de47a9883146409380d36199af27ca86"
I20260619 10:26:13.562233 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ca314916a44625bbcdc7611eab6661" candidate_uuid: "845247f61fda4017944ad2b02d61f41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd75a010eae04296b1d521538df3f06e"
I20260619 10:26:13.562314 13808 raft_consensus.cc:3037] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:13.562314 13941 raft_consensus.cc:3037] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:13.563202 13808 raft_consensus.cc:2454] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 845247f61fda4017944ad2b02d61f41f in term 1.
I20260619 10:26:13.563202 13941 raft_consensus.cc:2454] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 845247f61fda4017944ad2b02d61f41f in term 1.
I20260619 10:26:13.563350 14010 leader_election.cc:302] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 845247f61fda4017944ad2b02d61f41f, bd75a010eae04296b1d521538df3f06e; no voters:
I20260619 10:26:13.563432 14138 raft_consensus.cc:2781] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:13.563576 14138 raft_consensus.cc:686] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 1 LEADER]: Becoming Leader. State: Replica: 845247f61fda4017944ad2b02d61f41f, State: Running, Role: LEADER
I20260619 10:26:13.563666 14138 consensus_queue.cc:237] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } }
I20260619 10:26:13.564441 13664 catalog_manager.cc:5495] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f reported cstate change: term changed from 0 to 1, leader changed from <none> to 845247f61fda4017944ad2b02d61f41f (127.9.90.67). New cstate: current_term: 1 leader_uuid: "845247f61fda4017944ad2b02d61f41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845247f61fda4017944ad2b02d61f41f" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 } health_report { overall_health: UNKNOWN } } }
I20260619 10:26:13.581995 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.579903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.9.90.67:46719)
0619 10:26:13.579940 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.579942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.579953 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:26:13.579999 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.580000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.580016 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.580066 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:13.580161 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.580617 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.581289 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:13.581455 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:13.581497 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:13.581663 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:13.581757 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:13.581863 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.581899 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.581920 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:13.583595 13808 raft_consensus.cc:1261] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [term 1 FOLLOWER]: Refusing update from remote peer 845247f61fda4017944ad2b02d61f41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:13.583622 13941 raft_consensus.cc:1261] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [term 1 FOLLOWER]: Refusing update from remote peer 845247f61fda4017944ad2b02d61f41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:13.583890 14138 consensus_queue.cc:1035] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd75a010eae04296b1d521538df3f06e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:13.584022 14144 consensus_queue.cc:1035] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "de47a9883146409380d36199af27ca86" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:13.586133 14147 mvcc.cc:205] Tried to move back new op lower bound from 7298518112596209664 to 7298518112517414912. Current Snapshot: MvccSnapshot[applied={T|T < 7298518112596209664}]
I20260619 10:26:13.586724 14148 mvcc.cc:205] Tried to move back new op lower bound from 7298518112596209664 to 7298518112517414912. Current Snapshot: MvccSnapshot[applied={T|T < 7298518112596209664 or (T in {7298518112596209664})}]
I20260619 10:26:13.587229 14149 mvcc.cc:205] Tried to move back new op lower bound from 7298518112596209664 to 7298518112517414912. Current Snapshot: MvccSnapshot[applied={T|T < 7298518112596209664}]
I20260619 10:26:13.588507 13664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:13.588593 13664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:13.589288 13664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1f9482072704d90b062e298dad59fda: Sending DeleteTablet for 3 replicas of tablet d9ca314916a44625bbcdc7611eab6661
I20260619 10:26:13.589587 13788 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ca314916a44625bbcdc7611eab6661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:13 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:55226
I20260619 10:26:13.589634 13921 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ca314916a44625bbcdc7611eab6661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:13 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:59858
I20260619 10:26:13.589628 14054 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ca314916a44625bbcdc7611eab6661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:13 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:58524
I20260619 10:26:13.589864 14158 tablet_replica.cc:330] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: stopping tablet replica
I20260619 10:26:13.589896 14156 tablet_replica.cc:330] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e: stopping tablet replica
I20260619 10:26:13.589967 14158 raft_consensus.cc:2229] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:13.590034 14156 raft_consensus.cc:2229] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:13.590065 14158 raft_consensus.cc:2258] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:13.590204 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 13726
I20260619 10:26:13.590159 14156 raft_consensus.cc:2258] T d9ca314916a44625bbcdc7611eab6661 P bd75a010eae04296b1d521538df3f06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:13.590457 14158 ts_tablet_manager.cc:1905] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:13.591926 14158 ts_tablet_manager.cc:1918] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:13.591996 14158 log.cc:1196] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/d9ca314916a44625bbcdc7611eab6661
I20260619 10:26:13.592245 14158 ts_tablet_manager.cc:1939] T d9ca314916a44625bbcdc7611eab6661 P de47a9883146409380d36199af27ca86: Deleting consensus metadata
I20260619 10:26:13.592604 14157 tablet_replica.cc:330] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: stopping tablet replica
I20260619 10:26:13.592618 13651 catalog_manager.cc:4841] TS de47a9883146409380d36199af27ca86 (127.9.90.66:34429): tablet d9ca314916a44625bbcdc7611eab6661 (table test-table [id=11d313e3bf3a4aae8d4994c745df4b46]) successfully deleted
I20260619 10:26:13.592694 14157 raft_consensus.cc:2229] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:13.592787 14157 raft_consensus.cc:2258] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:13.593107 14157 ts_tablet_manager.cc:1905] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:13.594581 14157 ts_tablet_manager.cc:1918] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:13.594686 14157 log.cc:1196] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/d9ca314916a44625bbcdc7611eab6661
I20260619 10:26:13.594938 14157 ts_tablet_manager.cc:1939] T d9ca314916a44625bbcdc7611eab6661 P 845247f61fda4017944ad2b02d61f41f: Deleting consensus metadata
I20260619 10:26:13.595306 13648 catalog_manager.cc:4841] TS 845247f61fda4017944ad2b02d61f41f (127.9.90.67:46719): tablet d9ca314916a44625bbcdc7611eab6661 (table test-table [id=11d313e3bf3a4aae8d4994c745df4b46]) successfully deleted
W20260619 10:26:13.596645 13651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:43793 (error 108)
I20260619 10:26:13.597080 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.596961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:43793 (local address 127.0.0.1:55230)
0619 10:26:13.597014 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.597049 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 10:26:13.597091 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 13859
W20260619 10:26:13.598662 13651 catalog_manager.cc:4584] TS bd75a010eae04296b1d521538df3f06e (127.9.90.65:43793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9ca314916a44625bbcdc7611eab6661: Network error: Client connection negotiation failed: client connection to 127.9.90.65:43793: connect: Connection refused (error 111)
I20260619 10:26:13.601168 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 13992
I20260619 10:26:13.605020 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 13633
2026-06-19T10:26:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1830 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T10:26:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:13Z Disabled control of system clock
I20260619 10:26:13.631166 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42393
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 10:26:13.709184 14163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:13.709389 14163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:13.709408 14163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:13.710575 14163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:13.710619 14163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:13.710630 14163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:13.710641 14163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:13.710652 14163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:13.712002 14163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:42393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:13.712227 14163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:13.712522 14163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:13.717010 14169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.716984 14171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.716997 14168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:13.717288 14163 server_base.cc:1034] running on GCE node
I20260619 10:26:13.717496 14163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:13.717767 14163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:13.718930 14163 hybrid_clock.cc:648] HybridClock initialized: now 1781864773718918 us; error 29 us; skew 500 ppm
I20260619 10:26:13.720022 14163 webserver.cc:460] Webserver started at http://127.9.90.126:41473/ using document root <none> and password file <none>
I20260619 10:26:13.720230 14163 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:13.720279 14163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:13.720397 14163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:13.721366 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "3d746fc943fe4585ab00cd06b1ca4513"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "5e5d50ff78431ac490a5144b1b7cf357"
server_key_iv: "723a3f19d60f01514b3c095bdc359018"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.721661 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "3d746fc943fe4585ab00cd06b1ca4513"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "5e5d50ff78431ac490a5144b1b7cf357"
server_key_iv: "723a3f19d60f01514b3c095bdc359018"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.722995 14163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:13.723608 14177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.723754 14163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:13.723847 14163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "3d746fc943fe4585ab00cd06b1ca4513"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "5e5d50ff78431ac490a5144b1b7cf357"
server_key_iv: "723a3f19d60f01514b3c095bdc359018"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.723902 14163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:13.726994 14163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:13.727221 14163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:13.727331 14163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:13.730717 14163 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:42393
I20260619 10:26:13.730746 14229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:42393 every 8 connection(s)
I20260619 10:26:13.731015 14163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:13.731585 14230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:13.734164 14230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513: Bootstrap starting.
I20260619 10:26:13.734668 14230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:13.734814 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 14163
I20260619 10:26:13.734911 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:13.734944 14230 log.cc:824] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:13.735085 9577 external_mini_cluster.cc:1428] Setting key 74777ad5526930eeba8f3e613156d97d
I20260619 10:26:13.735594 14230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513: No bootstrap required, opened a new log
I20260619 10:26:13.737488 14230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d746fc943fe4585ab00cd06b1ca4513" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42393 } }
I20260619 10:26:13.737615 14230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:13.737638 14230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d746fc943fe4585ab00cd06b1ca4513, State: Initialized, Role: FOLLOWER
I20260619 10:26:13.737763 14230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d746fc943fe4585ab00cd06b1ca4513" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42393 } }
I20260619 10:26:13.737879 14230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:13.737914 14230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:13.737962 14230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:13.738106 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.735585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.9.90.126:42393)
0619 10:26:13.735790 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.735799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.735823 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:26:13.736712 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.736719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.736756 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.736883 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:13.737542 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:13.737545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:13.737545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:13.737549 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:13.737593 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:13.737602 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:13.737662 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.737860 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.737921 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260619 10:26:13.738533 14230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d746fc943fe4585ab00cd06b1ca4513" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42393 } }
I20260619 10:26:13.738622 14230 leader_election.cc:302] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d746fc943fe4585ab00cd06b1ca4513; no voters:
I20260619 10:26:13.738781 14230 leader_election.cc:288] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:13.738821 14235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:13.738972 14235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [term 1 LEADER]: Becoming Leader. State: Replica: 3d746fc943fe4585ab00cd06b1ca4513, State: Running, Role: LEADER
I20260619 10:26:13.739030 14230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:13.739076 14235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d746fc943fe4585ab00cd06b1ca4513" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42393 } }
I20260619 10:26:13.739461 14236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d746fc943fe4585ab00cd06b1ca4513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d746fc943fe4585ab00cd06b1ca4513" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42393 } } }
I20260619 10:26:13.739564 14236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:13.739758 14243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:13.739452 14237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d746fc943fe4585ab00cd06b1ca4513. Latest consensus state: current_term: 1 leader_uuid: "3d746fc943fe4585ab00cd06b1ca4513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d746fc943fe4585ab00cd06b1ca4513" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42393 } } }
I20260619 10:26:13.739993 14237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:13.740221 14243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:13.741498 14243 catalog_manager.cc:1347] Generated new cluster ID: bb5ca89e50c94401925f553d2905df42
I20260619 10:26:13.741540 14243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:13.746201 14243 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:13.746345 14243 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:13.754587 14243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513: Generated new TSK 0
I20260619 10:26:13.754755 14243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:26:13.762954 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42393
--builtin_ntp_servers=127.9.90.84:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 10:26:13.841360 14254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:13.841544 14254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:13.841572 14254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:13.842738 14254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:13.842777 14254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:13.842810 14254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:13.844096 14254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:13.844321 14254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:13.844575 14254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:13.849100 14262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.849140 14259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.849397 14260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:13.849344 14254 server_base.cc:1034] running on GCE node
I20260619 10:26:13.849642 14254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:13.849913 14254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:13.851051 14254 hybrid_clock.cc:648] HybridClock initialized: now 1781864773851037 us; error 37 us; skew 500 ppm
I20260619 10:26:13.852197 14254 webserver.cc:460] Webserver started at http://127.9.90.65:36905/ using document root <none> and password file <none>
I20260619 10:26:13.852413 14254 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:13.852461 14254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:13.852583 14254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:13.853554 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "d4305b7851d641f29bf8f4c7ed16968e"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "8a84828613e57296b45e28adeda1581f"
server_key_iv: "2925b2fe7347715f33ff539eb2c58d7f"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.853889 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "d4305b7851d641f29bf8f4c7ed16968e"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "8a84828613e57296b45e28adeda1581f"
server_key_iv: "2925b2fe7347715f33ff539eb2c58d7f"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.855080 14254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:13.855751 14268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.855903 14254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:13.855995 14254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "d4305b7851d641f29bf8f4c7ed16968e"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "8a84828613e57296b45e28adeda1581f"
server_key_iv: "2925b2fe7347715f33ff539eb2c58d7f"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.856060 14254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:13.860502 14254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:13.860761 14254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:13.860877 14254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:13.861068 14254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:13.861375 14254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:13.861415 14254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.861447 14254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:13.861469 14254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.867141 14254 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:42727
I20260619 10:26:13.867219 14381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:42727 every 8 connection(s)
I20260619 10:26:13.867480 14254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:13.870278 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.868120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:60531 (local address 127.9.90.126:42393)
0619 10:26:13.868170 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.868172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.868450 (+ 278us) server_negotiation.cc:407] Connection header received
0619 10:26:13.868883 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.868884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.868890 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.868941 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:13.870081 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:13.870082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:13.870083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:13.870087 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:13.870102 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:13.870106 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:13.870154 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.870216 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.870227 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:26:13.870450 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.868110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42393 (local address 127.9.90.65:60531)
0619 10:26:13.868294 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.868333 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.868798 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.869001 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.869004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.869012 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:13.869928 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:13.869929 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:13.869958 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:13.869987 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:13.870160 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:13.870162 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:13.870165 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.870216 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.870264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260619 10:26:13.870890 14382 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42393
I20260619 10:26:13.870970 14382 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:13.871104 14382 heartbeater.cc:511] Master 127.9.90.126:42393 requested a full tablet report, sending...
I20260619 10:26:13.871564 14194 ts_manager.cc:194] Registered new tserver with Master: d4305b7851d641f29bf8f4c7ed16968e (127.9.90.65:42727)
I20260619 10:26:13.877107 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 14254
I20260619 10:26:13.877189 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:13.877332 9577 external_mini_cluster.cc:1428] Setting key a0aea8ac39cf58bc9e740287c78b7235
I20260619 10:26:13.878146 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42393
--builtin_ntp_servers=127.9.90.84:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 10:26:13.956342 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:13.956543 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:13.956564 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:13.957657 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:13.957690 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:13.957712 14385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:13.958992 14385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:13.959193 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:13.959429 14385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:13.964201 14391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.964316 14390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:13.964215 14393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:13.964581 14385 server_base.cc:1034] running on GCE node
I20260619 10:26:13.964788 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:13.965032 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:13.966210 14385 hybrid_clock.cc:648] HybridClock initialized: now 1781864773966194 us; error 27 us; skew 500 ppm
I20260619 10:26:13.967379 14385 webserver.cc:460] Webserver started at http://127.9.90.66:42981/ using document root <none> and password file <none>
I20260619 10:26:13.967590 14385 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:13.967644 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:13.967772 14385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:13.968766 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "a828381ec55a424da357f27e3ae6973b"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "21b62792ec40108568679c21578c9576"
server_key_iv: "769ff54e53eec1b110d5fbb0a26ad273"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.969087 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "a828381ec55a424da357f27e3ae6973b"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "21b62792ec40108568679c21578c9576"
server_key_iv: "769ff54e53eec1b110d5fbb0a26ad273"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.970568 14385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:13.971241 14399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.971397 14385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:13.971475 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "a828381ec55a424da357f27e3ae6973b"
format_stamp: "Formatted at 2026-06-19 10:26:13 on dist-test-slave-hw7q"
server_key: "21b62792ec40108568679c21578c9576"
server_key_iv: "769ff54e53eec1b110d5fbb0a26ad273"
server_key_version: "encryptionkey@0"
I20260619 10:26:13.971540 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:13.974824 14385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:13.975063 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:13.975178 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:13.975366 14385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:13.975636 14385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:13.975667 14385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.975692 14385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:13.975716 14385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:13.981020 14385 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:39849
I20260619 10:26:13.981067 14512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:39849 every 8 connection(s)
I20260619 10:26:13.981364 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:13.982276 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 14385
I20260619 10:26:13.982365 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:13.982553 9577 external_mini_cluster.cc:1428] Setting key 0b9c0db8c66a3aaf424db60b7da6bf5c
I20260619 10:26:13.983448 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42393
--builtin_ntp_servers=127.9.90.84:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 10:26:13.984531 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.982034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:60349 (local address 127.9.90.126:42393)
0619 10:26:13.982081 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:13.982083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:13.982427 (+ 344us) server_negotiation.cc:407] Connection header received
0619 10:26:13.982981 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:13.982982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:13.982989 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:13.983040 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:13.984317 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:13.984318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:13.984319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:13.984322 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:13.984336 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:13.984340 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:13.984386 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:13.984466 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 10:26:13.984479 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:13.984745 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:13.982032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42393 (local address 127.9.90.66:60349)
0619 10:26:13.982281 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:13.982328 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:13.982879 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:13.983105 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:13.983108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:13.983115 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:13.984143 (+ 1028us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:13.984144 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:13.984180 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:13.984214 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:13.984397 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:13.984401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:13.984404 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:13.984476 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 10:26:13.984537 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260619 10:26:13.985145 14513 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42393
I20260619 10:26:13.985229 14513 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:13.985370 14513 heartbeater.cc:511] Master 127.9.90.126:42393 requested a full tablet report, sending...
I20260619 10:26:13.985697 14194 ts_manager.cc:194] Registered new tserver with Master: a828381ec55a424da357f27e3ae6973b (127.9.90.66:39849)
W20260619 10:26:14.062481 14516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:14.062683 14516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:14.062712 14516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:14.063849 14516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:14.063887 14516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:14.063920 14516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:14.065312 14516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:14.065553 14516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:14.065843 14516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:14.070461 14521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:14.070499 14522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:14.070640 14516 server_base.cc:1034] running on GCE node
W20260619 10:26:14.070683 14524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:14.070885 14516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:14.071096 14516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:14.072232 14516 hybrid_clock.cc:648] HybridClock initialized: now 1781864774072216 us; error 28 us; skew 500 ppm
I20260619 10:26:14.073336 14516 webserver.cc:460] Webserver started at http://127.9.90.67:40867/ using document root <none> and password file <none>
I20260619 10:26:14.073545 14516 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:14.073592 14516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:14.073714 14516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:14.074781 14516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "79786c129e234f998376a5ee1401e60b"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "262823c4e3b4cac92ebd5ae958c8686a"
server_key_iv: "1ed08f51531df843d618405443f25ba8"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.075129 14516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "79786c129e234f998376a5ee1401e60b"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "262823c4e3b4cac92ebd5ae958c8686a"
server_key_iv: "1ed08f51531df843d618405443f25ba8"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.076256 14516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:14.076870 14530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:14.077018 14516 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:14.077095 14516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "79786c129e234f998376a5ee1401e60b"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "262823c4e3b4cac92ebd5ae958c8686a"
server_key_iv: "1ed08f51531df843d618405443f25ba8"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.077159 14516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:14.080374 14516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:14.080605 14516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:14.080717 14516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:14.080905 14516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:14.081173 14516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:14.081204 14516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:14.081229 14516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:14.081252 14516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:14.086465 14516 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:34993
I20260619 10:26:14.086519 14643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:34993 every 8 connection(s)
I20260619 10:26:14.086794 14516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:26:14.088356 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 14516
I20260619 10:26:14.088443 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:14.088595 9577 external_mini_cluster.cc:1428] Setting key 0c0209eec99ee0e3049770c372e24240
I20260619 10:26:14.089574 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.087413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:57021 (local address 127.9.90.126:42393)
0619 10:26:14.087454 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.087456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.087671 (+ 215us) server_negotiation.cc:407] Connection header received
0619 10:26:14.088141 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.088143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.088150 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.088204 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.089400 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.089400 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.089401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.089404 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.089421 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.089425 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.089462 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.089515 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.089525 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:26:14.089756 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.087386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42393 (local address 127.9.90.67:57021)
0619 10:26:14.087568 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.087598 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.088046 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.088302 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.088308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.088315 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:14.089260 (+ 945us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:14.089261 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:14.089297 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:14.089331 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:14.089466 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:14.089469 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:14.089472 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:14.089522 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:26:14.089567 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260619 10:26:14.090119 14644 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42393
I20260619 10:26:14.090204 14644 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:14.090341 14644 heartbeater.cc:511] Master 127.9.90.126:42393 requested a full tablet report, sending...
I20260619 10:26:14.090658 14194 ts_manager.cc:194] Registered new tserver with Master: 79786c129e234f998376a5ee1401e60b (127.9.90.67:34993)
I20260619 10:26:14.100128 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:26:14.102249 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.101332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.9.90.126:42393)
0619 10:26:14.101375 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.101377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.101513 (+ 136us) server_negotiation.cc:407] Connection header received
0619 10:26:14.101554 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.101555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.101559 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.101601 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.102087 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.102088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.102089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.102092 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.102104 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.102108 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.102142 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.102172 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.102182 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:26:14.103600 14194 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:14.105031 14194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:14.108446 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.106911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:39849 (local address 127.0.0.1:35434)
0619 10:26:14.107074 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.107110 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.107162 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.107566 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.107571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.107576 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:14.108024 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:14.108025 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:14.108041 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:14.108062 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:14.108301 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:14.108307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:14.108309 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:26:14.108369 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 10:26:14.108381 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260619 10:26:14.108642 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.106993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.9.90.66:39849)
0619 10:26:14.107172 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.107177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.107186 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:26:14.107210 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.107212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.107221 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.107282 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.108137 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.108147 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.108147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.108151 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.108186 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.108216 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.108278 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.108545 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.108563 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260619 10:26:14.108934 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.107105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:42727 (local address 127.0.0.1:58584)
0619 10:26:14.107763 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.107771 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.107813 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.107993 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.107994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.107996 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:14.108362 (+ 366us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:14.108363 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:14.108369 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:14.108380 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:14.108821 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:14.108821 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:14.108822 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:26:14.108862 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 10:26:14.108875 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":67,"threads_started":1}
I20260619 10:26:14.109122 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.107330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.9.90.65:42727)
0619 10:26:14.107489 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.107492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.107873 (+ 381us) server_negotiation.cc:407] Connection header received
0619 10:26:14.107902 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.107905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.107914 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.107990 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.108640 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.108648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.108648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.108654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.108689 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.108717 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.108796 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.109032 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.109049 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260619 10:26:14.109297 14447 tablet_service.cc:1467] Processing CreateTablet for tablet fba20a6ad5a54cc8b679fe67044d23dd (DEFAULT_TABLE table=test-table [id=f7eaebcd5a254bb6a30fcc953fb6369b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:14.109606 14447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba20a6ad5a54cc8b679fe67044d23dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:14.109761 14316 tablet_service.cc:1467] Processing CreateTablet for tablet fba20a6ad5a54cc8b679fe67044d23dd (DEFAULT_TABLE table=test-table [id=f7eaebcd5a254bb6a30fcc953fb6369b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:14.110117 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba20a6ad5a54cc8b679fe67044d23dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:14.112267 14658 tablet_bootstrap.cc:492] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b: Bootstrap starting.
I20260619 10:26:14.112408 14659 tablet_bootstrap.cc:492] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e: Bootstrap starting.
I20260619 10:26:14.112478 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.107007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.9.90.67:34993)
0619 10:26:14.107919 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.107923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.107931 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:14.107954 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.107956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.107965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.108030 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.108734 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.108741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.108741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.108745 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.108786 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.108808 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.108863 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.112379 (+ 3516us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.112397 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":69,"threads_started":1}
I20260619 10:26:14.112691 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.106917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:34993 (local address 127.0.0.1:51592)
0619 10:26:14.107229 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.107239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.107284 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.108298 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.108299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.108301 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:14.108642 (+ 341us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:14.108643 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:14.108648 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:14.108660 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:14.108856 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:14.108857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:14.108858 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:26:14.112616 (+ 3758us) client_negotiation.cc:240] Negotiation successful
0619 10:26:14.112632 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":79,"threads_started":1}
I20260619 10:26:14.113039 14658 tablet_bootstrap.cc:654] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:14.113131 14659 tablet_bootstrap.cc:654] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:14.113336 14658 log.cc:824] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:14.113392 14659 log.cc:824] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:14.113390 14578 tablet_service.cc:1467] Processing CreateTablet for tablet fba20a6ad5a54cc8b679fe67044d23dd (DEFAULT_TABLE table=test-table [id=f7eaebcd5a254bb6a30fcc953fb6369b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:14.113685 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba20a6ad5a54cc8b679fe67044d23dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:14.114064 14658 tablet_bootstrap.cc:492] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b: No bootstrap required, opened a new log
I20260619 10:26:14.114118 14658 ts_tablet_manager.cc:1397] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:14.116493 14658 raft_consensus.cc:348] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } }
I20260619 10:26:14.116647 14658 raft_consensus.cc:374] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:14.116680 14658 raft_consensus.cc:729] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a828381ec55a424da357f27e3ae6973b, State: Initialized, Role: FOLLOWER
I20260619 10:26:14.116703 14662 tablet_bootstrap.cc:492] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: Bootstrap starting.
I20260619 10:26:14.116796 14658 consensus_queue.cc:260] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } }
I20260619 10:26:14.117027 14658 ts_tablet_manager.cc:1428] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:26:14.117076 14513 heartbeater.cc:503] Master 127.9.90.126:42393 was elected leader, sending a full tablet report...
I20260619 10:26:14.117332 14662 tablet_bootstrap.cc:654] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:14.117540 14659 tablet_bootstrap.cc:492] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e: No bootstrap required, opened a new log
I20260619 10:26:14.117609 14659 ts_tablet_manager.cc:1397] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 10:26:14.117621 14662 log.cc:824] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:14.118476 14662 tablet_bootstrap.cc:492] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: No bootstrap required, opened a new log
I20260619 10:26:14.118546 14662 ts_tablet_manager.cc:1397] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:14.120208 14659 raft_consensus.cc:348] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } }
I20260619 10:26:14.120380 14659 raft_consensus.cc:374] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:14.120414 14659 raft_consensus.cc:729] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4305b7851d641f29bf8f4c7ed16968e, State: Initialized, Role: FOLLOWER
I20260619 10:26:14.120551 14659 consensus_queue.cc:260] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } }
I20260619 10:26:14.120801 14659 ts_tablet_manager.cc:1428] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:26:14.120821 14382 heartbeater.cc:503] Master 127.9.90.126:42393 was elected leader, sending a full tablet report...
I20260619 10:26:14.120908 14662 raft_consensus.cc:348] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } }
I20260619 10:26:14.121040 14662 raft_consensus.cc:374] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:14.121079 14662 raft_consensus.cc:729] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79786c129e234f998376a5ee1401e60b, State: Initialized, Role: FOLLOWER
I20260619 10:26:14.121186 14662 consensus_queue.cc:260] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } }
I20260619 10:26:14.121443 14644 heartbeater.cc:503] Master 127.9.90.126:42393 was elected leader, sending a full tablet report...
I20260619 10:26:14.121428 14662 ts_tablet_manager.cc:1428] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:26:14.135437 14666 raft_consensus.cc:482] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:14.135535 14666 raft_consensus.cc:504] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } }
I20260619 10:26:14.135855 14666 leader_election.cc:288] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a828381ec55a424da357f27e3ae6973b (127.9.90.66:39849), d4305b7851d641f29bf8f4c7ed16968e (127.9.90.65:42727)
I20260619 10:26:14.137028 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.136000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:39849 (local address 127.9.90.67:45835)
0619 10:26:14.136064 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.136077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.136116 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.136287 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.136287 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.136289 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:14.136696 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:14.136697 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:14.136705 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:14.136718 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:14.136906 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:14.136907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:14.136907 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 10:26:14.136950 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:26:14.136963 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 10:26:14.137038 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.136156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:45835 (local address 127.9.90.66:39849)
0619 10:26:14.136197 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.136198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.136204 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:26:14.136222 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.136223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.136227 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.136275 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.136806 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.136808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.136808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.136813 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.136831 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.136836 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.136890 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.136960 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.136973 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:26:14.137125 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.136098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:55403 (local address 127.9.90.65:42727)
0619 10:26:14.136155 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.136157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.136245 (+ 88us) server_negotiation.cc:407] Connection header received
0619 10:26:14.136310 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.136311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.136316 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.136368 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.136925 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.136926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.136926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.136931 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.136947 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.136951 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.137002 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.137054 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.137065 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:26:14.137131 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.136005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:42727 (local address 127.9.90.67:55403)
0619 10:26:14.136188 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.136196 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.136244 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.136373 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.136374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.136376 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:14.136838 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:14.136838 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:14.136844 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:14.136854 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:14.137005 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:14.137006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:14.137007 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:26:14.137046 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 10:26:14.137057 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":11,"thread_start_us":102,"threads_started":1}
I20260619 10:26:14.137320 14467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba20a6ad5a54cc8b679fe67044d23dd" candidate_uuid: "79786c129e234f998376a5ee1401e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a828381ec55a424da357f27e3ae6973b" is_pre_election: true
I20260619 10:26:14.137393 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba20a6ad5a54cc8b679fe67044d23dd" candidate_uuid: "79786c129e234f998376a5ee1401e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4305b7851d641f29bf8f4c7ed16968e" is_pre_election: true
I20260619 10:26:14.137471 14467 raft_consensus.cc:2454] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79786c129e234f998376a5ee1401e60b in term 0.
I20260619 10:26:14.137512 14336 raft_consensus.cc:2454] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79786c129e234f998376a5ee1401e60b in term 0.
I20260619 10:26:14.137705 14532 leader_election.cc:302] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79786c129e234f998376a5ee1401e60b, d4305b7851d641f29bf8f4c7ed16968e; no voters:
I20260619 10:26:14.137841 14666 raft_consensus.cc:2781] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:14.137894 14666 raft_consensus.cc:482] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:14.137923 14666 raft_consensus.cc:3037] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:14.138644 14666 raft_consensus.cc:504] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } }
I20260619 10:26:14.138767 14666 leader_election.cc:288] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [CANDIDATE]: Term 1 election: Requested vote from peers a828381ec55a424da357f27e3ae6973b (127.9.90.66:39849), d4305b7851d641f29bf8f4c7ed16968e (127.9.90.65:42727)
I20260619 10:26:14.138901 14467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba20a6ad5a54cc8b679fe67044d23dd" candidate_uuid: "79786c129e234f998376a5ee1401e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a828381ec55a424da357f27e3ae6973b"
I20260619 10:26:14.138926 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba20a6ad5a54cc8b679fe67044d23dd" candidate_uuid: "79786c129e234f998376a5ee1401e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4305b7851d641f29bf8f4c7ed16968e"
I20260619 10:26:14.138989 14467 raft_consensus.cc:3037] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:14.139019 14336 raft_consensus.cc:3037] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:14.139850 14467 raft_consensus.cc:2454] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79786c129e234f998376a5ee1401e60b in term 1.
I20260619 10:26:14.139853 14336 raft_consensus.cc:2454] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79786c129e234f998376a5ee1401e60b in term 1.
I20260619 10:26:14.140030 14531 leader_election.cc:302] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79786c129e234f998376a5ee1401e60b, a828381ec55a424da357f27e3ae6973b; no voters:
I20260619 10:26:14.140134 14666 raft_consensus.cc:2781] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:14.140264 14666 raft_consensus.cc:686] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 1 LEADER]: Becoming Leader. State: Replica: 79786c129e234f998376a5ee1401e60b, State: Running, Role: LEADER
I20260619 10:26:14.140347 14666 consensus_queue.cc:237] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } }
I20260619 10:26:14.141094 14194 catalog_manager.cc:5495] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b reported cstate change: term changed from 0 to 1, leader changed from <none> to 79786c129e234f998376a5ee1401e60b (127.9.90.67). New cstate: current_term: 1 leader_uuid: "79786c129e234f998376a5ee1401e60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79786c129e234f998376a5ee1401e60b" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 } health_report { overall_health: UNKNOWN } } }
I20260619 10:26:14.144100 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.143378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.9.90.67:34993)
0619 10:26:14.143426 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.143428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.143460 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:26:14.143515 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.143515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.143524 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.143570 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.143958 (+ 388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.143959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.143959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.143962 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.143974 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.143977 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.144016 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.144048 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.144057 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:26:14.145546 14467 raft_consensus.cc:1261] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [term 1 FOLLOWER]: Refusing update from remote peer 79786c129e234f998376a5ee1401e60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:14.145593 14336 raft_consensus.cc:1261] T fba20a6ad5a54cc8b679fe67044d23dd P d4305b7851d641f29bf8f4c7ed16968e [term 1 FOLLOWER]: Refusing update from remote peer 79786c129e234f998376a5ee1401e60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:14.146008 14666 consensus_queue.cc:1035] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4305b7851d641f29bf8f4c7ed16968e" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:14.146123 14668 consensus_queue.cc:1035] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a828381ec55a424da357f27e3ae6973b" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:14.148669 14672 mvcc.cc:205] Tried to move back new op lower bound from 7298518114898120704 to 7298518114879528960. Current Snapshot: MvccSnapshot[applied={T|T < 7298518114898120704}]
I20260619 10:26:14.148805 14673 mvcc.cc:205] Tried to move back new op lower bound from 7298518114898120704 to 7298518114879528960. Current Snapshot: MvccSnapshot[applied={T|T < 7298518114898120704}]
I20260619 10:26:14.150142 14194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:14.150224 14194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:14.150852 14194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d746fc943fe4585ab00cd06b1ca4513: Sending DeleteTablet for 3 replicas of tablet fba20a6ad5a54cc8b679fe67044d23dd
I20260619 10:26:14.151146 14447 tablet_service.cc:1514] Processing DeleteTablet for tablet fba20a6ad5a54cc8b679fe67044d23dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:14 UTC) from {username='slave'} at 127.0.0.1:35434
I20260619 10:26:14.151356 14681 tablet_replica.cc:330] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b: stopping tablet replica
I20260619 10:26:14.151372 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 14254
I20260619 10:26:14.152078 14578 tablet_service.cc:1514] Processing DeleteTablet for tablet fba20a6ad5a54cc8b679fe67044d23dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:14 UTC) from {username='slave'} at 127.0.0.1:51592
W20260619 10:26:14.156742 14179 connection.cc:535] client connection to 127.9.90.65:42727 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:26:14.156754 14681 raft_consensus.cc:2229] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 10:26:14.156838 14179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:26:14.156858 14681 raft_consensus.cc:2258] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:14.156999 14682 tablet_replica.cc:330] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: stopping tablet replica
I20260619 10:26:14.157111 14682 raft_consensus.cc:2229] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:14.157274 14682 raft_consensus.cc:2258] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:14.157334 14681 ts_tablet_manager.cc:1905] T fba20a6ad5a54cc8b679fe67044d23dd P a828381ec55a424da357f27e3ae6973b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:14.157375 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 14385
I20260619 10:26:14.158897 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.158779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:42727 (local address 127.0.0.1:58588)
0619 10:26:14.158822 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.158864 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 10:26:14.159686 14179 catalog_manager.cc:4584] TS d4305b7851d641f29bf8f4c7ed16968e (127.9.90.65:42727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fba20a6ad5a54cc8b679fe67044d23dd: Network error: Client connection negotiation failed: client connection to 127.9.90.65:42727: connect: Connection refused (error 111)
I20260619 10:26:14.160323 14682 ts_tablet_manager.cc:1905] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:14.162132 14682 ts_tablet_manager.cc:1918] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:14.162212 14682 log.cc:1196] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/fba20a6ad5a54cc8b679fe67044d23dd
I20260619 10:26:14.162341 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 14516
I20260619 10:26:14.162562 14682 ts_tablet_manager.cc:1939] T fba20a6ad5a54cc8b679fe67044d23dd P 79786c129e234f998376a5ee1401e60b: Deleting consensus metadata
I20260619 10:26:14.162962 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.162532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:39849 (local address 127.0.0.1:35444)
0619 10:26:14.162906 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.162935 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.66:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355}
I20260619 10:26:14.166235 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 14163
I20260619 10:26:14.166247 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.166155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:34993 (local address 127.0.0.1:51598)
0619 10:26:14.166200 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.166225 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.67:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
2026-06-19T10:26:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (559 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 10:26:14.184085 9577 mini_oidc.cc:152] Starting JWKS server
I20260619 10:26:14.185091 9577 webserver.cc:460] Webserver started at https://127.0.0.1:43629/ using document root <none> and password file <none>
I20260619 10:26:14.185180 9577 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 10:26:14.185334 9577 webserver.cc:460] Webserver started at http://127.0.0.1:44441/ using document root <none> and password file <none>
2026-06-19T10:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:14Z Disabled control of system clock
I20260619 10:26:14.191956 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:37535
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:37535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43629/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/testchainca.cert with env {}
W20260619 10:26:14.269244 14689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:14.269435 14689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:14.269456 14689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:14.270583 14689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:14.270622 14689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:14.270638 14689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:14.270649 14689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:14.270660 14689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 10:26:14.270670 14689 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 10:26:14.270680 14689 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43629/jwks.json
I20260619 10:26:14.272115 14689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:37535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:37535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43629/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:14.272302 14689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:14.272449 14689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:14.276880 14694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:14.277029 14689 server_base.cc:1034] running on GCE node
W20260619 10:26:14.276832 14695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:14.276880 14697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:14.277328 14689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:14.277594 14689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:14.278736 14689 hybrid_clock.cc:648] HybridClock initialized: now 1781864774278711 us; error 42 us; skew 500 ppm
I20260619 10:26:14.279888 14689 webserver.cc:460] Webserver started at http://127.9.90.126:43337/ using document root <none> and password file <none>
I20260619 10:26:14.280097 14689 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:14.280143 14689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:14.280269 14689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:14.281256 14689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "1e24dc0af514482b884c3c2885228adf"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "bf7f5e49e612556e9ebaf6369eda26ea"
server_key_iv: "613245a36c3498e8ac2fad0c8b11606c"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.281567 14689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "1e24dc0af514482b884c3c2885228adf"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "bf7f5e49e612556e9ebaf6369eda26ea"
server_key_iv: "613245a36c3498e8ac2fad0c8b11606c"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.282799 14689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:14.283411 14703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:14.283560 14689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:14.283638 14689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "1e24dc0af514482b884c3c2885228adf"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "bf7f5e49e612556e9ebaf6369eda26ea"
server_key_iv: "613245a36c3498e8ac2fad0c8b11606c"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.283701 14689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:14.317463 14689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:14.317855 14689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:14.318012 14689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:14.322227 14689 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:37535
I20260619 10:26:14.322275 14755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:37535 every 8 connection(s)
I20260619 10:26:14.322604 14689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:14.323704 14756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:14.325950 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 14689
I20260619 10:26:14.326028 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:14.326198 9577 external_mini_cluster.cc:1428] Setting key 95557463cc387f44b490dc1cb4f00cc0
I20260619 10:26:14.326789 14756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf: Bootstrap starting.
I20260619 10:26:14.327350 14756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:14.327659 14756 log.cc:824] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:14.328406 14756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf: No bootstrap required, opened a new log
I20260619 10:26:14.330772 14756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e24dc0af514482b884c3c2885228adf" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37535 } }
I20260619 10:26:14.330909 14756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:14.330940 14756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e24dc0af514482b884c3c2885228adf, State: Initialized, Role: FOLLOWER
I20260619 10:26:14.331058 14756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e24dc0af514482b884c3c2885228adf" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37535 } }
I20260619 10:26:14.331147 14756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:14.331167 14756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:14.331194 14756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:14.331652 14756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e24dc0af514482b884c3c2885228adf" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37535 } }
I20260619 10:26:14.331728 14756 leader_election.cc:302] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e24dc0af514482b884c3c2885228adf; no voters:
I20260619 10:26:14.331882 14756 leader_election.cc:288] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:14.331924 14761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:14.332015 14761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [term 1 LEADER]: Becoming Leader. State: Replica: 1e24dc0af514482b884c3c2885228adf, State: Running, Role: LEADER
I20260619 10:26:14.332103 14761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e24dc0af514482b884c3c2885228adf" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37535 } }
I20260619 10:26:14.332136 14756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:14.332393 14761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e24dc0af514482b884c3c2885228adf. Latest consensus state: current_term: 1 leader_uuid: "1e24dc0af514482b884c3c2885228adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e24dc0af514482b884c3c2885228adf" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37535 } } }
I20260619 10:26:14.332470 14761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:14.332406 14763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e24dc0af514482b884c3c2885228adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e24dc0af514482b884c3c2885228adf" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37535 } } }
I20260619 10:26:14.332633 14763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf [sys.catalog]: This master's current role is: LEADER
W20260619 10:26:14.333721 14777 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:26:14.333763 14777 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:26:14.333834 14766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:14.334003 14766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:14.335561 14766 catalog_manager.cc:1347] Generated new cluster ID: 091968be36b74ad2b37436c13147527d
I20260619 10:26:14.335620 14766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:14.338138 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.326669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.9.90.126:37535)
0619 10:26:14.326853 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.326860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.326892 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:26:14.327316 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.327322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.327381 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.327512 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.327722 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.328713 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.329467 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.329953 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.330075 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.331051 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.331055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.331055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.331244 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.331294 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.331309 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.331369 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.331629 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.337901 (+ 6272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260619 10:26:14.339056 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:26:14.344996 14766 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:14.345438 14766 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:14.349701 14766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e24dc0af514482b884c3c2885228adf: Generated new TSK 0
I20260619 10:26:14.349911 14766 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 10:26:14.380702 14786 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 10:26:14.380796 14786 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 10:26:14.380931 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.358076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.9.90.126:37535)
0619 10:26:14.358114 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.358116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.358220 (+ 104us) server_negotiation.cc:407] Connection header received
0619 10:26:14.358308 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.358308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.358318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.358370 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0619 10:26:14.358494 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.358955 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.360042 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.360219 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.360260 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.360447 (+ 187us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 10:26:14.380714 (+ 20267us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 10:26:14.380852 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56398: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:26:14.380998 14759 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56398: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 10:26:14.381263 14785 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:14.357993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:37535 (local address 127.0.0.1:56398)
0619 10:26:14.358155 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.358166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.358232 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.358375 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.358376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.358378 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 10:26:14.358445 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:14.358447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.359032 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.359033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:14.359974 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:14.359976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.360275 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.360276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:14.360340 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.360361 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 10:26:14.380850 (+ 20489us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 10:26:14.380967 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.90.126:37535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.90.126:37535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 10:26:14.381639 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 14689
2026-06-19T10:26:14Z chronyd exiting
I20260619 10:26:14.416376 9577 test_util.cc:175] -----------------------------------------------
I20260619 10:26:14.416447 9577 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781864754199919-9577-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 10:26:14.417114 9577 mini_oidc.cc:152] Starting JWKS server
I20260619 10:26:14.417732 9577 webserver.cc:460] Webserver started at https://127.0.0.1:33367/ using document root <none> and password file <none>
I20260619 10:26:14.417779 9577 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 10:26:14.417943 9577 webserver.cc:460] Webserver started at http://127.0.0.1:34325/ using document root <none> and password file <none>
2026-06-19T10:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:14Z Disabled control of system clock
I20260619 10:26:14.424448 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:41455
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:41455
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33367/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/testchainca.cert with env {}
W20260619 10:26:14.500357 14792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:14.500553 14792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:14.500582 14792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:14.501881 14792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:14.501922 14792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:14.501940 14792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:14.501955 14792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:14.501971 14792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 10:26:14.501991 14792 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 10:26:14.502007 14792 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33367/jwks.json
I20260619 10:26:14.503491 14792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:41455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:41455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33367/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:14.503728 14792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:14.503924 14792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:14.508337 14800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:14.508323 14797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:14.508316 14798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:14.508549 14792 server_base.cc:1034] running on GCE node
I20260619 10:26:14.508759 14792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:14.509001 14792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:14.510138 14792 hybrid_clock.cc:648] HybridClock initialized: now 1781864774510122 us; error 28 us; skew 500 ppm
I20260619 10:26:14.511216 14792 webserver.cc:460] Webserver started at http://127.9.90.126:46599/ using document root <none> and password file <none>
I20260619 10:26:14.511422 14792 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:14.511469 14792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:14.511592 14792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:14.512573 14792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "fcd980f012104234a201e98cb2d32559"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "0701a69add541e4cf1a0fce2d51feeb7"
server_key_iv: "9b2feb8f8a384044a39c27d1e0590ce0"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.512897 14792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "fcd980f012104234a201e98cb2d32559"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "0701a69add541e4cf1a0fce2d51feeb7"
server_key_iv: "9b2feb8f8a384044a39c27d1e0590ce0"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.514045 14792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:14.514740 14806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:14.514891 14792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:14.514969 14792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "fcd980f012104234a201e98cb2d32559"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "0701a69add541e4cf1a0fce2d51feeb7"
server_key_iv: "9b2feb8f8a384044a39c27d1e0590ce0"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.515024 14792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:14.529285 14792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:14.529539 14792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:14.529656 14792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:14.533258 14792 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:41455
I20260619 10:26:14.533332 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:41455 every 8 connection(s)
I20260619 10:26:14.533608 14792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:14.534570 14859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:14.536716 14859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559: Bootstrap starting.
I20260619 10:26:14.537225 14859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:14.537436 14859 log.cc:824] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:14.537813 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 14792
I20260619 10:26:14.537904 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:14.538079 9577 external_mini_cluster.cc:1428] Setting key 2d2b8cb0f77e3466db8ad6c8ff35c49d
I20260619 10:26:14.538153 14859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559: No bootstrap required, opened a new log
I20260619 10:26:14.540508 14859 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd980f012104234a201e98cb2d32559" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41455 } }
I20260619 10:26:14.540639 14859 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:14.540670 14859 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd980f012104234a201e98cb2d32559, State: Initialized, Role: FOLLOWER
I20260619 10:26:14.540810 14859 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd980f012104234a201e98cb2d32559" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41455 } }
I20260619 10:26:14.540887 14859 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:14.540917 14859 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:14.540962 14859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:14.541477 14859 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd980f012104234a201e98cb2d32559" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41455 } }
I20260619 10:26:14.541575 14859 leader_election.cc:302] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcd980f012104234a201e98cb2d32559; no voters:
I20260619 10:26:14.541716 14859 leader_election.cc:288] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:14.541755 14864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:14.541914 14864 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [term 1 LEADER]: Becoming Leader. State: Replica: fcd980f012104234a201e98cb2d32559, State: Running, Role: LEADER
I20260619 10:26:14.541983 14859 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:14.542018 14864 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd980f012104234a201e98cb2d32559" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41455 } }
I20260619 10:26:14.542316 14865 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcd980f012104234a201e98cb2d32559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd980f012104234a201e98cb2d32559" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41455 } } }
I20260619 10:26:14.542387 14865 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:14.542560 14866 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcd980f012104234a201e98cb2d32559. Latest consensus state: current_term: 1 leader_uuid: "fcd980f012104234a201e98cb2d32559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd980f012104234a201e98cb2d32559" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41455 } } }
I20260619 10:26:14.542585 14870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:14.542611 14866 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:14.543102 14870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:14.543888 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.538496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.9.90.126:41455)
0619 10:26:14.538681 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.538688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.538711 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:26:14.539338 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.539345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.539400 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.539506 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.539725 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.540598 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.541470 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.542115 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.542276 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.543174 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.543178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.543178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.543375 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.543417 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.543431 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.543485 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.543601 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.543679 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260619 10:26:14.544884 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:26:14.544914 14870 catalog_manager.cc:1347] Generated new cluster ID: 27e7c2e67b7b490a9971114d63d96695
I20260619 10:26:14.544955 14870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:14.553339 14870 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:14.554036 14870 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:14.557456 14870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcd980f012104234a201e98cb2d32559: Generated new TSK 0
I20260619 10:26:14.557623 14870 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 10:26:14.585655 14889 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 10:26:14.585750 14889 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 10:26:14.586028 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.565231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.9.90.126:41455)
0619 10:26:14.565276 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.565278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.565436 (+ 158us) server_negotiation.cc:407] Connection header received
0619 10:26:14.565502 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.565503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.565516 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.565573 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0619 10:26:14.565686 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.566228 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.566969 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.567179 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.567232 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.567386 (+ 154us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 10:26:14.585814 (+ 18428us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 10:26:14.585965 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45352: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 10:26:14.586078 14863 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45352: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 10:26:14.586164 14888 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:14.565154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:41455 (local address 127.0.0.1:45352)
0619 10:26:14.565358 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.565368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.565437 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.565576 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.565577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.565579 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 10:26:14.565635 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:14.565636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.566312 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.566313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:14.566893 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:14.566895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.567239 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.567240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:14.567303 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.567308 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 10:26:14.585928 (+ 18620us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 10:26:14.586061 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.90.126:41455: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260619 10:26:14.586457 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 14792
2026-06-19T10:26:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 10:26:14.623132 9577 mini_oidc.cc:152] Starting JWKS server
I20260619 10:26:14.623775 9577 webserver.cc:460] Webserver started at https://127.0.0.1:44177/ using document root <none> and password file <none>
I20260619 10:26:14.623818 9577 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 10:26:14.623929 9577 webserver.cc:460] Webserver started at http://127.0.0.1:41311/ using document root <none> and password file <none>
2026-06-19T10:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:14Z Disabled control of system clock
I20260619 10:26:14.630784 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:44427
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:44427
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44177/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 10:26:14.707540 14895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:14.707732 14895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:14.707762 14895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:14.708837 14895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:14.708870 14895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:14.708882 14895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:14.708894 14895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:14.708904 14895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 10:26:14.708914 14895 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 10:26:14.708925 14895 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44177/jwks.json
I20260619 10:26:14.710209 14895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:44427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:44427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44177/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:14.710386 14895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:14.710605 14895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:26:14.714977 14895 server_base.cc:1034] running on GCE node
W20260619 10:26:14.714999 14901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:14.715073 14900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:14.714999 14903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:14.715374 14895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:14.715628 14895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:14.716765 14895 hybrid_clock.cc:648] HybridClock initialized: now 1781864774716749 us; error 28 us; skew 500 ppm
I20260619 10:26:14.717881 14895 webserver.cc:460] Webserver started at http://127.9.90.126:36643/ using document root <none> and password file <none>
I20260619 10:26:14.718079 14895 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:14.718125 14895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:14.718267 14895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:14.719240 14895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "6258047543cc4e5493f0ed27eba19ffe"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "3bc310721d64608dc78c14c33835f13d"
server_key_iv: "13898491793277fb77bd6c40d962d81a"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.719556 14895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "6258047543cc4e5493f0ed27eba19ffe"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "3bc310721d64608dc78c14c33835f13d"
server_key_iv: "13898491793277fb77bd6c40d962d81a"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.720731 14895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:14.721410 14909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:14.721567 14895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:14.721644 14895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "6258047543cc4e5493f0ed27eba19ffe"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "3bc310721d64608dc78c14c33835f13d"
server_key_iv: "13898491793277fb77bd6c40d962d81a"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.721702 14895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:14.748405 14895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:14.748723 14895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:14.748852 14895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:14.752377 14895 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:44427
I20260619 10:26:14.752434 14961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:44427 every 8 connection(s)
I20260619 10:26:14.752743 14895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:14.753382 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:14.754714 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 14895
I20260619 10:26:14.754805 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:14.754976 9577 external_mini_cluster.cc:1428] Setting key 11e93a58374e4aa7eda63ee9121fdb17
I20260619 10:26:14.756343 14962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe: Bootstrap starting.
I20260619 10:26:14.756956 14962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:14.757190 14962 log.cc:824] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:14.757833 14962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe: No bootstrap required, opened a new log
I20260619 10:26:14.759443 14962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6258047543cc4e5493f0ed27eba19ffe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44427 } }
I20260619 10:26:14.759573 14962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:14.759605 14962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6258047543cc4e5493f0ed27eba19ffe, State: Initialized, Role: FOLLOWER
I20260619 10:26:14.759728 14962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6258047543cc4e5493f0ed27eba19ffe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44427 } }
I20260619 10:26:14.759812 14962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:14.759842 14962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:14.759889 14962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:14.760128 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.755431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.9.90.126:44427)
0619 10:26:14.755593 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.755599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.755623 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:26:14.756163 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.756172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.756235 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.756350 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.756565 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.757490 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.758317 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.758735 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.758840 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.759437 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.759440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.759440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.759587 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.759620 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.759644 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.759703 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.759805 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.759983 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260619 10:26:14.760432 14962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6258047543cc4e5493f0ed27eba19ffe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44427 } }
I20260619 10:26:14.760516 14962 leader_election.cc:302] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6258047543cc4e5493f0ed27eba19ffe; no voters:
I20260619 10:26:14.760669 14962 leader_election.cc:288] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:14.760697 14967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:14.760805 14967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [term 1 LEADER]: Becoming Leader. State: Replica: 6258047543cc4e5493f0ed27eba19ffe, State: Running, Role: LEADER
I20260619 10:26:14.760877 14967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6258047543cc4e5493f0ed27eba19ffe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44427 } }
I20260619 10:26:14.760905 14962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:14.760946 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:26:14.761176 14967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6258047543cc4e5493f0ed27eba19ffe. Latest consensus state: current_term: 1 leader_uuid: "6258047543cc4e5493f0ed27eba19ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6258047543cc4e5493f0ed27eba19ffe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44427 } } }
I20260619 10:26:14.761245 14967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:14.761617 14968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6258047543cc4e5493f0ed27eba19ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6258047543cc4e5493f0ed27eba19ffe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44427 } } }
I20260619 10:26:14.761698 14968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:14.761974 14972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:14.762399 14972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:14.763762 14972 catalog_manager.cc:1347] Generated new cluster ID: 9c0fb06c188f455cb31cc8cc4510eb39
I20260619 10:26:14.763809 14972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:14.774999 14972 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:14.775676 14972 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:14.780028 14972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6258047543cc4e5493f0ed27eba19ffe: Generated new TSK 0
I20260619 10:26:14.780191 14972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:26:14.794474 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.790788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.9.90.126:44427)
0619 10:26:14.790841 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.790842 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.790915 (+ 73us) server_negotiation.cc:407] Connection header received
0619 10:26:14.790968 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.790969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.790981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.791058 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0619 10:26:14.791143 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.791935 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.792635 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.792980 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.793045 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.793273 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 10:26:14.794273 (+ 1000us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 10:26:14.794407 (+ 134us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41518: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 10:26:14.794533 14965 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41518: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 10:26:14.794835 14991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:14.790722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:44427 (local address 127.0.0.1:41518)
0619 10:26:14.790860 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.790868 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.790919 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.791056 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.791056 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.791058 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 10:26:14.791108 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:14.791110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.791999 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.791999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:14.792569 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:14.792571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.793075 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.793076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:14.793156 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.793162 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 10:26:14.794626 (+ 1464us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 10:26:14.794722 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.90.126:44427: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260619 10:26:14.795174 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 14895
E20260619 10:26:14.805914 14992 webserver.cc:533] Webserver: sslize failed:
E20260619 10:26:14.805979 14992 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T10:26:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (208 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T10:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:14Z Disabled control of system clock
I20260619 10:26:14.838099 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:38841
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:38841
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/testchainca.cert with env {}
W20260619 10:26:14.915395 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:14.915589 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:14.915609 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:14.916754 14996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:14.916801 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:14.916812 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:14.916826 14996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:14.916848 14996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 10:26:14.916872 14996 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 10:26:14.916893 14996 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 10:26:14.918349 14996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:38841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:38841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:14.918568 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:14.918756 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:14.923161 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:14.923187 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:14.923229 14996 server_base.cc:1034] running on GCE node
W20260619 10:26:14.923170 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:14.923563 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:14.923823 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:14.924983 14996 hybrid_clock.cc:648] HybridClock initialized: now 1781864774924971 us; error 26 us; skew 500 ppm
I20260619 10:26:14.926120 14996 webserver.cc:460] Webserver started at http://127.9.90.126:37947/ using document root <none> and password file <none>
I20260619 10:26:14.926328 14996 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:14.926374 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:14.926488 14996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:14.927435 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "bb65522d87cc4e8f98143d9300fbf30d"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "0d596a00e7d4b5768721d7b9a6ed07f8"
server_key_iv: "2c64a76b1cff300f78de10b9e7985f20"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.927755 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "bb65522d87cc4e8f98143d9300fbf30d"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "0d596a00e7d4b5768721d7b9a6ed07f8"
server_key_iv: "2c64a76b1cff300f78de10b9e7985f20"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.928838 14996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:14.929416 15010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:14.929561 14996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:14.929638 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "bb65522d87cc4e8f98143d9300fbf30d"
format_stamp: "Formatted at 2026-06-19 10:26:14 on dist-test-slave-hw7q"
server_key: "0d596a00e7d4b5768721d7b9a6ed07f8"
server_key_iv: "2c64a76b1cff300f78de10b9e7985f20"
server_key_version: "encryptionkey@0"
I20260619 10:26:14.929697 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:14.938535 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:14.938793 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:14.938910 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:14.942472 14996 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:38841
I20260619 10:26:14.942502 15062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:38841 every 8 connection(s)
I20260619 10:26:14.942795 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:14.943621 15063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:14.945730 15063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d: Bootstrap starting.
I20260619 10:26:14.946313 15063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:14.946553 15063 log.cc:824] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:14.947146 15063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d: No bootstrap required, opened a new log
I20260619 10:26:14.948648 15063 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb65522d87cc4e8f98143d9300fbf30d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38841 } }
I20260619 10:26:14.948796 15063 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:14.948820 15063 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb65522d87cc4e8f98143d9300fbf30d, State: Initialized, Role: FOLLOWER
I20260619 10:26:14.948980 15063 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb65522d87cc4e8f98143d9300fbf30d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38841 } }
I20260619 10:26:14.949057 15063 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:14.949092 15063 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:14.949138 15063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:14.949630 15063 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb65522d87cc4e8f98143d9300fbf30d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38841 } }
I20260619 10:26:14.949724 15063 leader_election.cc:302] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb65522d87cc4e8f98143d9300fbf30d; no voters:
I20260619 10:26:14.949906 15063 leader_election.cc:288] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:14.949954 15066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:14.950085 15066 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [term 1 LEADER]: Becoming Leader. State: Replica: bb65522d87cc4e8f98143d9300fbf30d, State: Running, Role: LEADER
I20260619 10:26:14.950145 15063 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:14.950176 15066 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb65522d87cc4e8f98143d9300fbf30d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38841 } }
I20260619 10:26:14.950532 15068 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb65522d87cc4e8f98143d9300fbf30d. Latest consensus state: current_term: 1 leader_uuid: "bb65522d87cc4e8f98143d9300fbf30d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb65522d87cc4e8f98143d9300fbf30d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38841 } } }
I20260619 10:26:14.950634 15067 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb65522d87cc4e8f98143d9300fbf30d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb65522d87cc4e8f98143d9300fbf30d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38841 } } }
I20260619 10:26:14.950711 15067 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:14.950670 15068 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:14.951089 15075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:14.951550 15075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:14.951613 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 14996
I20260619 10:26:14.951710 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:14.951898 9577 external_mini_cluster.cc:1428] Setting key 2773402acdfe9f5cad0bfd938cc72dd2
I20260619 10:26:14.953493 15075 catalog_manager.cc:1347] Generated new cluster ID: c378e7e2fdcd41e491222b32b29098f2
I20260619 10:26:14.953544 15075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:14.962085 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.952358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.9.90.126:38841)
0619 10:26:14.952675 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.952682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.952705 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:26:14.952792 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.952797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.952847 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.952959 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:14.953151 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.958921 (+ 5770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.960041 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.960602 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.960710 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.961376 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:14.961380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:14.961380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:14.961573 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:14.961613 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:14.961627 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:14.961682 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:14.961755 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 10:26:14.961864 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":67,"threads_started":1}
I20260619 10:26:14.962890 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:26:14.970084 15075 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:14.970772 15075 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:14.973529 15075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb65522d87cc4e8f98143d9300fbf30d: Generated new TSK 0
I20260619 10:26:14.973678 15075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:26:15.422787 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:14.984828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.9.90.126:38841)
0619 10:26:14.984883 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:14.984885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:14.984985 (+ 100us) server_negotiation.cc:407] Connection header received
0619 10:26:14.985055 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:14.985056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:14.985065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:14.985119 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0619 10:26:14.985223 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.985712 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.986815 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.987005 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.987051 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.987204 (+ 153us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 10:26:15.422497 (+435293us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 10:26:15.422697 (+ 200us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50300: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 10:26:15.422876 15085 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50300: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 10:26:15.422878 15092 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:14.984762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:38841 (local address 127.0.0.1:50300)
0619 10:26:14.984920 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:14.984931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:14.984997 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:14.985120 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:14.985121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:14.985123 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 10:26:14.985176 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:14.985178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.985780 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.985781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:14.986744 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:14.986746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:14.987065 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:14.987066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:14.987130 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:14.987135 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 10:26:15.422654 (+435519us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 10:26:15.422780 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.90.126:38841: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260619 10:26:15.423285 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 14996
2026-06-19T10:26:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (608 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (83 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (82 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:15 dist-test-slave-hw7q krb5kdc[15104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:15 dist-test-slave-hw7q krb5kdc[15104](info): set up 2 sockets
Jun 19 10:26:15 dist-test-slave-hw7q krb5kdc[15104](info): commencing operation
krb5kdc: starting...
W20260619 10:26:16.537654 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:16 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:16.646081 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:41161
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:41161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:16.723821 15117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:16.724023 15117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:16.724053 15117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:16.725188 15117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:16.725226 15117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:16.725252 15117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:16.725268 15117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:16.725284 15117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:16.726724 15117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:41161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:16.726989 15117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:16.727191 15117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:16.731526 15123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:16.731505 15122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:16.731534 15117 server_base.cc:1034] running on GCE node
W20260619 10:26:16.731518 15125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:16.731921 15117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:16.732174 15117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:16.733310 15117 hybrid_clock.cc:648] HybridClock initialized: now 1781864776733300 us; error 32 us; skew 500 ppm
Jun 19 10:26:16 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864776, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:16.740571 15117 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:16.741044 15117 webserver.cc:460] Webserver started at http://127.9.90.126:46191/ using document root <none> and password file <none>
I20260619 10:26:16.741250 15117 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:16.741298 15117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:16.741422 15117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:16.742491 15117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "7fbe9d2a8542476b839d34388c94e6fe"
format_stamp: "Formatted at 2026-06-19 10:26:16 on dist-test-slave-hw7q"
server_key: "b54471cee23ea454861fa1d8a484472f"
server_key_iv: "86a5b96b9e8b8debd5ecf3c739ed90dd"
server_key_version: "encryptionkey@0"
I20260619 10:26:16.742825 15117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "7fbe9d2a8542476b839d34388c94e6fe"
format_stamp: "Formatted at 2026-06-19 10:26:16 on dist-test-slave-hw7q"
server_key: "b54471cee23ea454861fa1d8a484472f"
server_key_iv: "86a5b96b9e8b8debd5ecf3c739ed90dd"
server_key_version: "encryptionkey@0"
I20260619 10:26:16.744038 15117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:26:16.744685 15132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:16.744843 15117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:16.744925 15117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "7fbe9d2a8542476b839d34388c94e6fe"
format_stamp: "Formatted at 2026-06-19 10:26:16 on dist-test-slave-hw7q"
server_key: "b54471cee23ea454861fa1d8a484472f"
server_key_iv: "86a5b96b9e8b8debd5ecf3c739ed90dd"
server_key_version: "encryptionkey@0"
I20260619 10:26:16.744982 15117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:16.783447 15117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:16.785557 15117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:16.785692 15117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:16.789395 15117 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:41161
I20260619 10:26:16.789453 15184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:41161 every 8 connection(s)
I20260619 10:26:16.789772 15117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:16.790617 15185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:16.791114 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 15117
I20260619 10:26:16.791226 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:16.791427 9577 external_mini_cluster.cc:1428] Setting key 9f6e5be4c8148e7eac358bf28eae6d05
I20260619 10:26:16.793282 15185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe: Bootstrap starting.
I20260619 10:26:16.793879 15185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:16.794129 15185 log.cc:824] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:16.794895 15185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe: No bootstrap required, opened a new log
I20260619 10:26:16.797124 15185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbe9d2a8542476b839d34388c94e6fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41161 } }
I20260619 10:26:16.797290 15185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:16.797324 15185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbe9d2a8542476b839d34388c94e6fe, State: Initialized, Role: FOLLOWER
I20260619 10:26:16.797456 15185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbe9d2a8542476b839d34388c94e6fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41161 } }
I20260619 10:26:16.797542 15185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:16.797567 15185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:16.797614 15185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [term 0 FOLLOWER]: Advancing to term 1
Jun 19 10:26:16 dist-test-slave-hw7q krb5kdc[15104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:16.798209 15185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbe9d2a8542476b839d34388c94e6fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41161 } }
I20260619 10:26:16.798292 15185 leader_election.cc:302] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fbe9d2a8542476b839d34388c94e6fe; no voters:
I20260619 10:26:16.798437 15185 leader_election.cc:288] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:16.798465 15190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:16.798589 15190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [term 1 LEADER]: Becoming Leader. State: Replica: 7fbe9d2a8542476b839d34388c94e6fe, State: Running, Role: LEADER
I20260619 10:26:16.798699 15185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:16.798683 15190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbe9d2a8542476b839d34388c94e6fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41161 } }
I20260619 10:26:16.799072 15191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fbe9d2a8542476b839d34388c94e6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbe9d2a8542476b839d34388c94e6fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41161 } } }
I20260619 10:26:16.799487 15191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:16.799214 15192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fbe9d2a8542476b839d34388c94e6fe. Latest consensus state: current_term: 1 leader_uuid: "7fbe9d2a8542476b839d34388c94e6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbe9d2a8542476b839d34388c94e6fe" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41161 } } }
I20260619 10:26:16.799774 15192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:16.799921 15201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:16.802628 15201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:16.803568 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:16.791950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.9.90.126:41161)
0619 10:26:16.792123 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:16.792131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:16.792155 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:26:16.792535 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:16.792541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:16.792586 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:16.792711 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:16.793322 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:16.794391 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:16.795470 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:16.795876 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:16.795955 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:16.798473 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:16.798477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:16.798478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:16.798489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:16.801586 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:16.802002 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:16.802005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:16.802006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:16.802113 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:16.802345 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:16.802347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:16.802347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:16.802668 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:16.802817 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:16.803321 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 10:26:16.803428 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260619 10:26:16.804294 15201 catalog_manager.cc:1347] Generated new cluster ID: ee004fda62c34889a7514b85cb65be0d
I20260619 10:26:16.804335 15201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:16.827831 15201 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:16.828612 15201 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:16.832065 15201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fbe9d2a8542476b839d34388c94e6fe: Generated new TSK 0
I20260619 10:26:16.832242 15201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:26:16.854161 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:41161
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:16.930675 15211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:16.930867 15211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:16.930888 15211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:16.931985 15211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:16.932024 15211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:16.932052 15211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:16.933454 15211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:41161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:16.933663 15211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:16.933884 15211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:16.938416 15216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:16.938417 15219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:16.938521 15211 server_base.cc:1034] running on GCE node
W20260619 10:26:16.938458 15217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:16.938819 15211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:16.939038 15211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:16.940171 15211 hybrid_clock.cc:648] HybridClock initialized: now 1781864776940159 us; error 29 us; skew 500 ppm
Jun 19 10:26:16 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864776, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:16.942502 15211 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:26:16.942912 15211 webserver.cc:460] Webserver started at http://127.9.90.65:39305/ using document root <none> and password file <none>
I20260619 10:26:16.943094 15211 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:16.943138 15211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:16.943259 15211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:16.944224 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "d49d1179ecee44599489aa4a0ee49c5f"
format_stamp: "Formatted at 2026-06-19 10:26:16 on dist-test-slave-hw7q"
server_key: "fb72f197f55d7b0f69fb662c9fd0b3df"
server_key_iv: "c7b4c1c9543f3389ec0b1df43a30c390"
server_key_version: "encryptionkey@0"
I20260619 10:26:16.944533 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "d49d1179ecee44599489aa4a0ee49c5f"
format_stamp: "Formatted at 2026-06-19 10:26:16 on dist-test-slave-hw7q"
server_key: "fb72f197f55d7b0f69fb662c9fd0b3df"
server_key_iv: "c7b4c1c9543f3389ec0b1df43a30c390"
server_key_version: "encryptionkey@0"
I20260619 10:26:16.945664 15211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:16.946350 15226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:16.946496 15211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:16.946573 15211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "d49d1179ecee44599489aa4a0ee49c5f"
format_stamp: "Formatted at 2026-06-19 10:26:16 on dist-test-slave-hw7q"
server_key: "fb72f197f55d7b0f69fb662c9fd0b3df"
server_key_iv: "c7b4c1c9543f3389ec0b1df43a30c390"
server_key_version: "encryptionkey@0"
I20260619 10:26:16.946637 15211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:16.959403 15211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:16.961339 15211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:16.961465 15211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:16.961668 15211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:16.962003 15211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:16.962037 15211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:16.962065 15211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:16.962090 15211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:16.967410 15211 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:41191
I20260619 10:26:16.967466 15339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:41191 every 8 connection(s)
I20260619 10:26:16.967756 15211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:16.968475 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 15211
I20260619 10:26:16.968570 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:16.968788 9577 external_mini_cluster.cc:1428] Setting key d158dbbddf77512543d14c06b5fa99f5
Jun 19 10:26:16 dist-test-slave-hw7q krb5kdc[15104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864776, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
I20260619 10:26:16.977502 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:16.968441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:55515 (local address 127.9.90.126:41161)
0619 10:26:16.968505 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:16.968507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:16.968756 (+ 249us) server_negotiation.cc:407] Connection header received
0619 10:26:16.969830 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:16.969831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:16.969842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:16.969882 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:16.970718 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:16.971447 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:16.972459 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:16.972770 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:16.972843 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:16.974893 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:16.974895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:16.974895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:16.974905 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:16.976325 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:16.976690 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:16.976691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:16.976692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:16.976762 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:16.976962 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:16.976963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:16.976963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:16.977068 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:16.977118 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:16.977359 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 10:26:16.977436 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:26:16.977702 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:16.968415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:41161 (local address 127.9.90.65:55515)
0619 10:26:16.968619 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:16.968668 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:16.969710 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:16.969961 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:16.969965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:16.970318 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:16.970636 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:16.970642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:16.971558 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:16.971559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:16.972375 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:16.972377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:16.972851 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:16.972852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:16.972955 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:16.973454 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:16.973462 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:16.974803 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:16.976408 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:16.976413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:16.976415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:16.976620 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:16.976822 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:16.976823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:16.976823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:16.976916 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:16.977113 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:16.977116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:16.977258 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 10:26:16.977338 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 10:26:16.977476 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260619 10:26:16.978214 15340 heartbeater.cc:347] Connected to a master server at 127.9.90.126:41161
I20260619 10:26:16.978299 15340 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:16.978463 15340 heartbeater.cc:511] Master 127.9.90.126:41161 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
I20260619 10:26:16.978951 15149 ts_manager.cc:194] Registered new tserver with Master: d49d1179ecee44599489aa4a0ee49c5f (127.9.90.65:41191)
I20260619 10:26:16.979784 15149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:55515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:26:16.990464 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:41161
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:17.069954 15345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:17.070156 15345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:17.070186 15345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:17.071367 15345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:17.071409 15345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:17.071444 15345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:17.072865 15345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:41161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:17.073112 15345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:17.073328 15345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:17.078133 15353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:17.078181 15351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:17.078145 15350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:17.078444 15345 server_base.cc:1034] running on GCE node
I20260619 10:26:17.078658 15345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:17.078902 15345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:17.080094 15345 hybrid_clock.cc:648] HybridClock initialized: now 1781864777080076 us; error 34 us; skew 500 ppm
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:17.082566 15345 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:26:17.082993 15345 webserver.cc:460] Webserver started at http://127.9.90.66:45997/ using document root <none> and password file <none>
I20260619 10:26:17.083204 15345 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:17.083250 15345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:17.083371 15345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:17.084393 15345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "f2ff4e6915a14b5598019a8d08981b78"
format_stamp: "Formatted at 2026-06-19 10:26:17 on dist-test-slave-hw7q"
server_key: "d09336fbe30294547499aefd06b53691"
server_key_iv: "4c58e826a7fe2315ab1fe9e61cf1e492"
server_key_version: "encryptionkey@0"
I20260619 10:26:17.084707 15345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "f2ff4e6915a14b5598019a8d08981b78"
format_stamp: "Formatted at 2026-06-19 10:26:17 on dist-test-slave-hw7q"
server_key: "d09336fbe30294547499aefd06b53691"
server_key_iv: "4c58e826a7fe2315ab1fe9e61cf1e492"
server_key_version: "encryptionkey@0"
I20260619 10:26:17.086024 15345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:26:17.086764 15360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:17.086935 15345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:17.087014 15345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "f2ff4e6915a14b5598019a8d08981b78"
format_stamp: "Formatted at 2026-06-19 10:26:17 on dist-test-slave-hw7q"
server_key: "d09336fbe30294547499aefd06b53691"
server_key_iv: "4c58e826a7fe2315ab1fe9e61cf1e492"
server_key_version: "encryptionkey@0"
I20260619 10:26:17.087074 15345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:17.102313 15345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:17.103994 15345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:17.104127 15345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:17.104337 15345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:17.104615 15345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:17.104647 15345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:17.104676 15345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:17.104692 15345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:17.110009 15345 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:35115
I20260619 10:26:17.110060 15473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:35115 every 8 connection(s)
I20260619 10:26:17.110379 15345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:17.114506 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 15345
I20260619 10:26:17.114612 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:17.114810 9577 external_mini_cluster.cc:1428] Setting key fab91cd1c928be7e5eb384d72c9f1cbb
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:17.119107 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:17.111063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:36053 (local address 127.9.90.126:41161)
0619 10:26:17.111110 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:17.111113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:17.111323 (+ 210us) server_negotiation.cc:407] Connection header received
0619 10:26:17.112172 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:17.112173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:17.112182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:17.112228 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:17.112782 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.113264 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.114182 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.114366 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.114410 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:17.116542 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:17.116543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:17.116544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:17.116553 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:17.117864 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.118221 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.118222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.118222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.118292 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.118496 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.118497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.118497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.118615 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:17.118671 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:17.118924 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 10:26:17.119019 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:26:17.119243 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:17.111044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:41161 (local address 127.9.90.66:36053)
0619 10:26:17.111193 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:17.111227 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:17.112097 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:17.112297 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:17.112300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:17.112540 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:17.112725 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:17.112729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.113325 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.113325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:17.114125 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:17.114127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.114432 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.114433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:17.114577 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:17.115062 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:17.115072 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:17.116450 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:17.117925 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:17.117930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:17.117931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:17.118143 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:17.118344 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:17.118345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:17.118345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:17.118441 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:17.118668 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:17.118674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:17.118810 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 10:26:17.118894 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 10:26:17.119039 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260619 10:26:17.119650 15474 heartbeater.cc:347] Connected to a master server at 127.9.90.126:41161
I20260619 10:26:17.119732 15474 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:17.119894 15474 heartbeater.cc:511] Master 127.9.90.126:41161 requested a full tablet report, sending...
I20260619 10:26:17.120249 15149 ts_manager.cc:194] Registered new tserver with Master: f2ff4e6915a14b5598019a8d08981b78 (127.9.90.66:35115)
I20260619 10:26:17.120594 15149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:36053
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:26:17.136585 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:41161
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:17.216352 15479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:17.216542 15479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:17.216565 15479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:17.217674 15479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:17.217712 15479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:17.217741 15479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:17.219194 15479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:41161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:17.219422 15479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:17.219619 15479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:17.224347 15484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:17.224557 15485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:17.224609 15487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:17.224661 15479 server_base.cc:1034] running on GCE node
I20260619 10:26:17.224874 15479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:17.225101 15479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:17.226229 15479 hybrid_clock.cc:648] HybridClock initialized: now 1781864777226215 us; error 34 us; skew 500 ppm
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:17.228524 15479 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:26:17.228915 15479 webserver.cc:460] Webserver started at http://127.9.90.67:35603/ using document root <none> and password file <none>
I20260619 10:26:17.229118 15479 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:17.229164 15479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:17.229290 15479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:17.230324 15479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "740f5e6e971e496f91064ba2a63c81a3"
format_stamp: "Formatted at 2026-06-19 10:26:17 on dist-test-slave-hw7q"
server_key: "c1f8c3ec6cde507cf501ff70e6325fb2"
server_key_iv: "93565be8235ea661cd7e4deea26e7f89"
server_key_version: "encryptionkey@0"
I20260619 10:26:17.230635 15479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "740f5e6e971e496f91064ba2a63c81a3"
format_stamp: "Formatted at 2026-06-19 10:26:17 on dist-test-slave-hw7q"
server_key: "c1f8c3ec6cde507cf501ff70e6325fb2"
server_key_iv: "93565be8235ea661cd7e4deea26e7f89"
server_key_version: "encryptionkey@0"
I20260619 10:26:17.231822 15479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:17.232440 15494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:17.232590 15479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:17.232667 15479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "740f5e6e971e496f91064ba2a63c81a3"
format_stamp: "Formatted at 2026-06-19 10:26:17 on dist-test-slave-hw7q"
server_key: "c1f8c3ec6cde507cf501ff70e6325fb2"
server_key_iv: "93565be8235ea661cd7e4deea26e7f89"
server_key_version: "encryptionkey@0"
I20260619 10:26:17.232726 15479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:17.252311 15479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:17.254382 15479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:17.254516 15479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:17.254726 15479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:17.255010 15479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:17.255043 15479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:17.255071 15479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:17.255092 15479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:17.260329 15479 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:34311
I20260619 10:26:17.260383 15607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:34311 every 8 connection(s)
I20260619 10:26:17.260659 15479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:26:17.262032 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 15479
I20260619 10:26:17.262120 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:17.262276 9577 external_mini_cluster.cc:1428] Setting key ebd2e9c646f47a56df2bd55acc187598
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:17.268965 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:17.261277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:37727 (local address 127.9.90.126:41161)
0619 10:26:17.261317 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:17.261318 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:17.261516 (+ 198us) server_negotiation.cc:407] Connection header received
0619 10:26:17.262545 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:17.262546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:17.262558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:17.262604 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:17.263183 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.263677 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.264631 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.264813 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.264868 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:17.266742 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:17.266743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:17.266743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:17.266751 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:17.267799 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.268148 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.268149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.268149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.268199 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.268404 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.268405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.268406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.268506 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:17.268559 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:17.268805 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 10:26:17.268877 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:26:17.269122 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:17.261259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:41161 (local address 127.9.90.67:37727)
0619 10:26:17.261402 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:17.261438 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:17.262459 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:17.262681 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:17.262685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:17.262925 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:17.263102 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:17.263106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.263764 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.263765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:17.264553 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:17.264555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.264886 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.264887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:17.265006 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:17.265408 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:17.265416 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:17.266662 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:17.267885 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:17.267891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:17.267893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:17.268084 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:17.268251 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:17.268252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:17.268252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:17.268351 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:17.268558 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:17.268564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:17.268687 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 10:26:17.268777 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 10:26:17.268918 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260619 10:26:17.269569 15608 heartbeater.cc:347] Connected to a master server at 127.9.90.126:41161
I20260619 10:26:17.269654 15608 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:17.269819 15608 heartbeater.cc:511] Master 127.9.90.126:41161 requested a full tablet report, sending...
I20260619 10:26:17.270154 15149 ts_manager.cc:194] Registered new tserver with Master: 740f5e6e971e496f91064ba2a63c81a3 (127.9.90.67:34311)
I20260619 10:26:17.270498 15149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:37727
I20260619 10:26:17.273810 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/test-admin.keytab.
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:17.287731 9577 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 10:26:17.289273 15615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:17.288609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.9.90.65:41191)
0619 10:26:17.288788 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:17.288793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:17.288802 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:26:17.288826 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:17.288828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:17.288841 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:17.288904 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:17.289195 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47886: BlockingRecv error: recv got EOF from 127.0.0.1:47886 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
W20260619 10:26:17.289469 9577 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5cc' of type FILE
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:17.290438 9577 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:17.291246 9577 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:26:17.298141 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:17.291722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.9.90.65:41191)
0619 10:26:17.291775 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:17.291776 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:17.291809 (+ 33us) server_negotiation.cc:407] Connection header received
0619 10:26:17.291864 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:17.291864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:17.291874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:17.291927 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:17.292252 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.292946 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.293608 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.293960 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.294009 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:17.295845 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:17.295851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:17.295851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:17.295857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:17.297095 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.297409 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.297411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.297412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.297464 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.297622 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.297623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.297623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.297752 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:17.297889 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:17.297949 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 10:26:17.298043 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 10:26:17.299638 15615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:17.299209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.9.90.65:41191)
0619 10:26:17.299258 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:17.299260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:17.299315 (+ 55us) server_negotiation.cc:407] Connection header received
0619 10:26:17.299358 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:17.299358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:17.299366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:17.299405 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:17.299605 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47912: BlockingRecv error: recv got EOF from 127.0.0.1:47912 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 10:26:17.299840 9577 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5cc' of type FILE
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:17.300681 9577 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:17.301407 9577 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:26:17.307466 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:17.301869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.9.90.65:41191)
0619 10:26:17.301914 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:17.301916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:17.301925 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:26:17.301979 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:17.301979 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:17.301985 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:17.302018 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:17.302364 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.302901 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.303598 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.303807 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.303860 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:17.305601 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:17.305602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:17.305602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:17.305608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:17.306674 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.306953 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.306953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.306954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.306999 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.307123 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.307123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.307124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.307200 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:17.307275 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:17.307348 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 10:26:17.307405 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 10:26:17.308840 15615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:17.308369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.9.90.65:41191)
0619 10:26:17.308406 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:17.308408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:17.308415 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:17.308472 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:17.308473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:17.308480 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:17.308517 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:17.308793 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47916: BlockingRecv error: recv got EOF from 127.0.0.1:47916 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 10:26:17.308914 9577 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781864754199919-9577-0/krb5kdc/krb5cc' of type FILE
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:17.309733 9577 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:17.310552 9577 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:26:17.316092 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:17.310972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.9.90.65:41191)
0619 10:26:17.311019 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:17.311021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:17.311031 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:17.311075 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:17.311076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:17.311082 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:17.311121 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:17.311399 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.311837 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.312432 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:17.312597 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:17.312635 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:17.314369 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:17.314370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:17.314370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:17.314376 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:17.315365 (+ 989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.315621 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.315622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.315622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.315662 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:17.315784 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:17.315785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:17.315785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:17.315859 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:17.315916 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:17.315990 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 10:26:17.316036 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:26:17.316622 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 15211
I20260619 10:26:17.320379 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 15345
I20260619 10:26:17.324023 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 15479
I20260619 10:26:17.327071 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 15117
2026-06-19T10:26:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1738 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15628](info): set up 2 sockets
Jun 19 10:26:17 dist-test-slave-hw7q krb5kdc[15628](info): commencing operation
krb5kdc: starting...
W20260619 10:26:18.277750 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:18 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.126.keytab.
Entry for principal oryx/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.126.keytab.
I20260619 10:26:18.382406 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33979
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.126.keytab
--principal=oryx/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:33979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:18.459398 15641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:18.459609 15641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:18.459640 15641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:18.460747 15641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:18.460786 15641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:18.460804 15641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:18.460820 15641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:18.460836 15641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:18.462312 15641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:33979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.126.keytab
--principal=oryx/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:18.462541 15641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:18.462738 15641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:18.467077 15649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:18.467064 15646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:18.467067 15647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:18.467345 15641 server_base.cc:1034] running on GCE node
I20260619 10:26:18.467573 15641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:18.467832 15641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:18.468976 15641 hybrid_clock.cc:648] HybridClock initialized: now 1781864778468961 us; error 25 us; skew 500 ppm
Jun 19 10:26:18 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:18.471369 15641 init.cc:377] Logged in from keytab as oryx/127.9.90.126@KRBTEST.COM (short username oryx)
I20260619 10:26:18.471808 15641 webserver.cc:460] Webserver started at http://127.9.90.126:45815/ using document root <none> and password file <none>
I20260619 10:26:18.471999 15641 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:18.472047 15641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:18.472170 15641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:18.473196 15641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "e9b360c5f4e3437d8dee672b25e2dba4"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "b3c1dab01d8fc644dd7d4deb02249543"
server_key_iv: "14e73c709baef45e9837ad339fe79d82"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.473552 15641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "e9b360c5f4e3437d8dee672b25e2dba4"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "b3c1dab01d8fc644dd7d4deb02249543"
server_key_iv: "14e73c709baef45e9837ad339fe79d82"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.474807 15641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:18.475498 15656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.475649 15641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:18.475729 15641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "e9b360c5f4e3437d8dee672b25e2dba4"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "b3c1dab01d8fc644dd7d4deb02249543"
server_key_iv: "14e73c709baef45e9837ad339fe79d82"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.475796 15641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:18.484505 15641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:18.486307 15641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:18.486428 15641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:18.490013 15641 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:33979
I20260619 10:26:18.490042 15708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:33979 every 8 connection(s)
I20260619 10:26:18.490358 15641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:18.491261 15709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:18.493398 15709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: Bootstrap starting.
I20260619 10:26:18.493965 15709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:18.494184 15709 log.cc:824] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:18.494793 15709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: No bootstrap required, opened a new log
I20260619 10:26:18.496311 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 15641
I20260619 10:26:18.496428 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:18.496379 15709 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } }
I20260619 10:26:18.496546 15709 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:18.496585 15709 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b360c5f4e3437d8dee672b25e2dba4, State: Initialized, Role: FOLLOWER
I20260619 10:26:18.496596 9577 external_mini_cluster.cc:1428] Setting key 99ebf09a37a5ec6ef75767c1280ebf69
I20260619 10:26:18.496727 15709 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } }
I20260619 10:26:18.496810 15709 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:18.496832 15709 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:18.496865 15709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:18.497582 15709 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } }
I20260619 10:26:18.497694 15709 leader_election.cc:302] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b360c5f4e3437d8dee672b25e2dba4; no voters:
I20260619 10:26:18.497864 15709 leader_election.cc:288] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:18.497892 15714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:18.498018 15714 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 1 LEADER]: Becoming Leader. State: Replica: e9b360c5f4e3437d8dee672b25e2dba4, State: Running, Role: LEADER
I20260619 10:26:18.498102 15714 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } }
I20260619 10:26:18.498131 15709 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:18.498476 15715 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } }
I20260619 10:26:18.498553 15715 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:18.498690 15716 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b360c5f4e3437d8dee672b25e2dba4. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } }
I20260619 10:26:18.498764 15716 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:18.498876 15720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:18.499581 15720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:18.501049 15720 catalog_manager.cc:1347] Generated new cluster ID: d1f382c22a204f1b8f634281311475a0
I20260619 10:26:18.501098 15720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:26:18 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:18.508985 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:18.497229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.9.90.126:33979)
0619 10:26:18.497774 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:18.497801 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:18.497842 (+ 41us) server_negotiation.cc:407] Connection header received
0619 10:26:18.497964 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:18.497972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:18.498033 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:18.498991 (+ 958us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:18.499099 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.500534 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.501590 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.502241 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.502340 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:18.504952 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:18.504956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:18.504957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:18.504969 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:18.507436 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:18.507786 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:18.507790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:18.507790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:18.507908 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:18.508083 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:18.508084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:18.508084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:18.508392 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:18.508545 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:18.508613 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 10:26:18.508763 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":427,"threads_started":1}
I20260619 10:26:18.519727 15720 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:18.520418 15720 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:18.523561 15720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: Generated new TSK 0
I20260619 10:26:18.523715 15720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab.
Entry for principal oryx/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab.
I20260619 10:26:18.548470 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33979
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab
--principal=oryx/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:18.625844 15735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:18.626039 15735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:18.626060 15735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:18.627162 15735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:18.627202 15735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:18.627226 15735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:18.628630 15735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab
--principal=oryx/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:18.628855 15735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:18.629060 15735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:18.633647 15741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:18.633659 15740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:18.633773 15735 server_base.cc:1034] running on GCE node
W20260619 10:26:18.633733 15743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:18.634033 15735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:18.634258 15735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:18.635401 15735 hybrid_clock.cc:648] HybridClock initialized: now 1781864778635391 us; error 30 us; skew 500 ppm
Jun 19 10:26:18 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:18.637648 15735 init.cc:377] Logged in from keytab as oryx/127.9.90.65@KRBTEST.COM (short username oryx)
I20260619 10:26:18.638067 15735 webserver.cc:460] Webserver started at http://127.9.90.65:42787/ using document root <none> and password file <none>
I20260619 10:26:18.638273 15735 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:18.638327 15735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:18.638445 15735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:18.639443 15735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "83c8715c390f407da0848f3d08206803"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "b98ffa148769409d7a0c6d35b8a449fd"
server_key_iv: "9bc41f1e790ab15704f43eab8619b366"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.639766 15735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "83c8715c390f407da0848f3d08206803"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "b98ffa148769409d7a0c6d35b8a449fd"
server_key_iv: "9bc41f1e790ab15704f43eab8619b366"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.640934 15735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:18.641623 15750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.641808 15735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.641892 15735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "83c8715c390f407da0848f3d08206803"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "b98ffa148769409d7a0c6d35b8a449fd"
server_key_iv: "9bc41f1e790ab15704f43eab8619b366"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.641954 15735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:18.658037 15735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:18.659894 15735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:18.660004 15735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:18.660169 15735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:18.660440 15735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:18.660470 15735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.660491 15735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:18.660504 15735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.665887 15735 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:37485
I20260619 10:26:18.665969 15863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:37485 every 8 connection(s)
I20260619 10:26:18.666254 15735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:26:18 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:18.672922 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 15735
I20260619 10:26:18.672991 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:18.673153 9577 external_mini_cluster.cc:1428] Setting key 93a5d03ead436ab75026471f928e63d7
I20260619 10:26:18.674813 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:18.666926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:58425 (local address 127.9.90.126:33979)
0619 10:26:18.666976 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:18.666978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:18.667142 (+ 164us) server_negotiation.cc:407] Connection header received
0619 10:26:18.667603 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:18.667604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:18.667613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:18.667667 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:18.668518 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.669010 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.669987 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.670171 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.670217 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:18.672152 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:18.672154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:18.672154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:18.672160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:18.673480 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:18.673918 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:18.673920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:18.673921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:18.673980 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:18.674198 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:18.674199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:18.674200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:18.674321 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:18.674376 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:18.674639 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 10:26:18.674724 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:26:18.675163 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:18.666885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.65:58425)
0619 10:26:18.667032 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:18.667072 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:18.667526 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:18.667723 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:18.667726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:18.668267 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:18.668449 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:18.668454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.669074 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.669074 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:18.669919 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:18.669921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.670227 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.670228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:18.670325 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:18.670756 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:18.670765 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:18.672059 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:18.673544 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:18.673550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:18.673552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:18.673848 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:18.674034 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:18.674035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:18.674035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:18.674144 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:18.674376 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:18.674383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:18.674519 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 10:26:18.674610 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 10:26:18.674776 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260619 10:26:18.675644 15864 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33979
I20260619 10:26:18.675731 15864 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:18.675908 15864 heartbeater.cc:511] Master 127.9.90.126:33979 requested a full tablet report, sending...
I20260619 10:26:18.676457 15673 ts_manager.cc:194] Registered new tserver with Master: 83c8715c390f407da0848f3d08206803 (127.9.90.65:37485)
I20260619 10:26:18.677332 15673 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:58425
WARNING: no policy specified for oryx/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab.
Entry for principal oryx/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab.
I20260619 10:26:18.694819 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33979
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab
--principal=oryx/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:18.770913 15869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:18.771101 15869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:18.771122 15869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:18.772168 15869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:18.772200 15869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:18.772233 15869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:18.773566 15869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab
--principal=oryx/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:18.773731 15869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:18.773947 15869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:18.778635 15875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:18.779748 15869 server_base.cc:1034] running on GCE node
W20260619 10:26:18.779846 15877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:18.780304 15874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:18.780515 15869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:18.780704 15869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:18.781845 15869 hybrid_clock.cc:648] HybridClock initialized: now 1781864778781827 us; error 27 us; skew 500 ppm
Jun 19 10:26:18 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:18.784032 15869 init.cc:377] Logged in from keytab as oryx/127.9.90.66@KRBTEST.COM (short username oryx)
I20260619 10:26:18.784443 15869 webserver.cc:460] Webserver started at http://127.9.90.66:46403/ using document root <none> and password file <none>
I20260619 10:26:18.784633 15869 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:18.784678 15869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:18.784803 15869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:18.785843 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "ad5e370ed46448198c12796120515e87"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "7ad18588a29112efde2fd07cb2784ff1"
server_key_iv: "3940355451a07384e69a8913d127cab2"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.786194 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "ad5e370ed46448198c12796120515e87"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "7ad18588a29112efde2fd07cb2784ff1"
server_key_iv: "3940355451a07384e69a8913d127cab2"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.787412 15869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:18.788105 15884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.788268 15869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:18.788347 15869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "ad5e370ed46448198c12796120515e87"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "7ad18588a29112efde2fd07cb2784ff1"
server_key_iv: "3940355451a07384e69a8913d127cab2"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.788414 15869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:18.810946 15869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:18.812674 15869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:18.812835 15869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:18.813056 15869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:18.813349 15869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:18.813380 15869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.813409 15869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:18.813432 15869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.818718 15869 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:45495
I20260619 10:26:18.818830 15997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:45495 every 8 connection(s)
I20260619 10:26:18.819083 15869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:26:18 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:18.827541 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:18.819717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:36201 (local address 127.9.90.126:33979)
0619 10:26:18.819768 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:18.819770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:18.819959 (+ 189us) server_negotiation.cc:407] Connection header received
0619 10:26:18.820470 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:18.820471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:18.820484 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:18.820537 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:18.821430 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.821981 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.822942 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.823147 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.823203 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:18.825168 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:18.825169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:18.825169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:18.825175 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:18.826437 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:18.826754 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:18.826755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:18.826755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:18.826803 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:18.826976 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:18.826976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:18.826976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:18.827066 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:18.827109 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:18.827393 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 10:26:18.827467 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:26:18.827574 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:18.819684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.66:36201)
0619 10:26:18.819845 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:18.819884 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:18.820375 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:18.820615 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:18.820619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:18.821160 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:18.821339 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:18.821344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.822070 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.822071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:18.822864 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:18.822866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.823214 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.823215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:18.823327 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:18.823778 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:18.823786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:18.825068 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:18.826503 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:18.826506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:18.826508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:18.826697 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:18.826849 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:18.826849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:18.826850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:18.826934 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:18.827105 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:18.827107 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:18.827217 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 10:26:18.827317 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 10:26:18.827424 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260619 10:26:18.827975 15998 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33979
I20260619 10:26:18.828094 15998 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:18.828263 15998 heartbeater.cc:511] Master 127.9.90.126:33979 requested a full tablet report, sending...
I20260619 10:26:18.828613 15673 ts_manager.cc:194] Registered new tserver with Master: ad5e370ed46448198c12796120515e87 (127.9.90.66:45495)
I20260619 10:26:18.828955 15673 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:36201
I20260619 10:26:18.829000 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 15869
I20260619 10:26:18.829092 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:18.829300 9577 external_mini_cluster.cc:1428] Setting key 50fbafa288bb38c5f405fa56985265db
WARNING: no policy specified for oryx/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab.
Entry for principal oryx/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab.
I20260619 10:26:18.849990 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33979
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab
--principal=oryx/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:18.927726 16003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:18.927928 16003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:18.927949 16003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:18.929106 16003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:18.929157 16003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:18.929183 16003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:18.930656 16003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab
--principal=oryx/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:18.930882 16003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:18.931090 16003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:18.935631 16011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:18.935714 16009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:18.935806 16003 server_base.cc:1034] running on GCE node
W20260619 10:26:18.935658 16008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:18.936024 16003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:18.936224 16003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:18.937356 16003 hybrid_clock.cc:648] HybridClock initialized: now 1781864778937331 us; error 40 us; skew 500 ppm
Jun 19 10:26:18 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:18.939687 16003 init.cc:377] Logged in from keytab as oryx/127.9.90.67@KRBTEST.COM (short username oryx)
I20260619 10:26:18.940132 16003 webserver.cc:460] Webserver started at http://127.9.90.67:33649/ using document root <none> and password file <none>
I20260619 10:26:18.940346 16003 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:18.940393 16003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:18.940516 16003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:18.941533 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "ad4bf6b0bc984d66856c3ad71c7ead83"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "6b099c1d25385aa99dcacb548fad407f"
server_key_iv: "05048ee8612b59655c982a5f88e8fbe1"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.941910 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "ad4bf6b0bc984d66856c3ad71c7ead83"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "6b099c1d25385aa99dcacb548fad407f"
server_key_iv: "05048ee8612b59655c982a5f88e8fbe1"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.943082 16003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:18.943735 16018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.943881 16003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:18.943959 16003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "ad4bf6b0bc984d66856c3ad71c7ead83"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "6b099c1d25385aa99dcacb548fad407f"
server_key_iv: "05048ee8612b59655c982a5f88e8fbe1"
server_key_version: "encryptionkey@0"
I20260619 10:26:18.944023 16003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:18.957454 16003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:18.959586 16003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:18.959726 16003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:18.959945 16003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:18.960242 16003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:18.960278 16003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.960307 16003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:18.960330 16003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:18.965651 16003 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:45345
I20260619 10:26:18.965711 16131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:45345 every 8 connection(s)
I20260619 10:26:18.966022 16003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:26:18 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:18.974078 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 16003
I20260619 10:26:18.974164 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:18.974339 9577 external_mini_cluster.cc:1428] Setting key 4123b6370f127083b7e0e17ea5876a55
I20260619 10:26:18.974565 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:18.966787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:53469 (local address 127.9.90.126:33979)
0619 10:26:18.966834 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:18.966836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:18.967030 (+ 194us) server_negotiation.cc:407] Connection header received
0619 10:26:18.967504 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:18.967506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:18.967516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:18.967555 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:18.968413 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.968890 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.969822 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.969995 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.970040 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:18.971957 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:18.971958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:18.971958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:18.971964 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:18.973160 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:18.973544 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:18.973545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:18.973545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:18.973599 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:18.973838 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:18.973839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:18.973840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:18.973965 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:18.974020 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:18.974358 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 10:26:18.974438 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:18.974675 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:18.966764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.67:53469)
0619 10:26:18.966920 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:18.966954 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:18.967411 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:18.967625 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:18.967628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:18.968161 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:18.968349 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:18.968354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.968951 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.968952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:18.969743 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:18.969745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:18.970048 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:18.970049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:18.970143 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:18.970544 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:18.970552 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:18.971866 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:18.973234 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:18.973241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:18.973243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:18.973475 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:18.973652 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:18.973653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:18.973653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:18.973756 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:18.974021 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:18.974028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:18.974200 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 10:26:18.974310 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 10:26:18.974458 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260619 10:26:18.975070 16132 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33979
I20260619 10:26:18.975167 16132 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:18.975342 16132 heartbeater.cc:511] Master 127.9.90.126:33979 requested a full tablet report, sending...
I20260619 10:26:18.975683 15673 ts_manager.cc:194] Registered new tserver with Master: ad4bf6b0bc984d66856c3ad71c7ead83 (127.9.90.67:45345)
I20260619 10:26:18.976011 15673 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:53469
I20260619 10:26:18.986480 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.90.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.125.keytab.
Entry for principal oryx/127.9.90.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.125.keytab.
I20260619 10:26:19.009089 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.125:46451
--webserver_interface=127.9.90.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.125.keytab
--principal=oryx/127.9.90.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:33979,127.9.90.125:46451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:19.088685 16137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:19.088899 16137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:19.088920 16137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:19.090098 16137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:19.090142 16137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:19.090153 16137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:19.090164 16137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:19.090175 16137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:19.091701 16137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:33979,127.9.90.125:46451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.125.keytab
--principal=oryx/127.9.90.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.125:46451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.90.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
W20260619 10:26:19.091806 16137 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.90.126:33979,127.9.90.125:46451'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 10:26:19.091955 16137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:19.092159 16137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:19.096639 16145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.096654 16142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.097707 16137 server_base.cc:1034] running on GCE node
W20260619 10:26:19.097884 16143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.098111 16137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:19.098358 16137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:19.099485 16137 hybrid_clock.cc:648] HybridClock initialized: now 1781864779099486 us; error 41 us; skew 500 ppm
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:19.101752 16137 init.cc:377] Logged in from keytab as oryx/127.9.90.125@KRBTEST.COM (short username oryx)
I20260619 10:26:19.102180 16137 webserver.cc:460] Webserver started at http://127.9.90.125:41201/ using document root <none> and password file <none>
I20260619 10:26:19.102375 16137 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:19.102421 16137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:19.102536 16137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:19.103543 16137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data/instance:
uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "262767d85aabb02759356c38f3c47f52"
server_key_iv: "3aa7b5c2a0a5c1c83e1299c37f3f468a"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.103875 16137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal/instance:
uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "262767d85aabb02759356c38f3c47f52"
server_key_iv: "3aa7b5c2a0a5c1c83e1299c37f3f468a"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.105043 16137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:26:19.105706 16152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.105878 16137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.105957 16137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal
uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "262767d85aabb02759356c38f3c47f52"
server_key_iv: "3aa7b5c2a0a5c1c83e1299c37f3f468a"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.106017 16137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:19.120328 16137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:19.122260 16137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:19.122392 16137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:19.125972 16137 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.125:46451
I20260619 10:26:19.126025 16204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.125:46451 every 8 connection(s)
I20260619 10:26:19.126332 16137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
I20260619 10:26:19.127301 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:19.128794 16205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:26:19.133944 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 16137
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.125@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:19.137202 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.129472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.9.90.126:33979)
0619 10:26:19.129518 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.129520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.129729 (+ 209us) server_negotiation.cc:407] Connection header received
0619 10:26:19.130249 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.130250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.130262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.130322 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.130913 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.131394 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.132348 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.132529 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.132582 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.134711 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.134712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.134713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.134719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.135947 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.136293 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.136295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.136295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.136354 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.136564 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.136565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.136566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.136711 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.136771 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.137022 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.137114 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:19.137343 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.129443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:47554)
0619 10:26:19.129616 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.129650 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.130149 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.130401 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.130405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.130644 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.130829 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.130833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.131471 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.131472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.132267 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.132270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.132597 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.132598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.132713 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.133162 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.133171 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.134641 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.136016 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.136022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.136023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.136222 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.136407 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.136408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.136408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.136510 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.136780 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.136785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.136910 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.136992 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.137131 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260619 10:26:19.137979 16205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.90.125@KRBTEST.COM
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.125@KRBTEST.COM for oryx/127.9.90.125@KRBTEST.COM
I20260619 10:26:19.141989 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.134393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.9.90.125:46451)
0619 10:26:19.134524 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.134528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.134566 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:26:19.134631 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.134633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.134643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.134711 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.135339 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.136075 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.136786 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.137072 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.137127 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.139471 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.139478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.139478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.139485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.140907 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.141219 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.141222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.141222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.141277 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.141434 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.141435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.141436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.141590 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.141722 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.141820 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.141894 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260619 10:26:19.144949 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.138237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.0.0.1:33954)
0619 10:26:19.138276 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.138286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.138318 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.138521 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.138521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.138618 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.138660 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.138661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.139313 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.139314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.139841 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.139843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.140529 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.140530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.140592 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.140975 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.140979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.142268 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.144103 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.144104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.144105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.144254 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.144428 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.144429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.144429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.144491 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.144705 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.144705 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.144747 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.144811 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.144887 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 10:26:19.145056 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.138308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.9.90.125:46451)
0619 10:26:19.138426 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.138428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.138433 (+ 5us) server_negotiation.cc:407] Connection header received
0619 10:26:19.138448 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.138448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.138456 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.138514 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.138729 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.139243 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.140161 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.140443 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.140513 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.142351 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.142352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.142352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.142357 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.144042 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.144309 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.144310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.144310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.144355 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.144544 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.144545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.144545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.144653 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.144697 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.144837 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.144931 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260619 10:26:19.146270 16205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Bootstrap starting.
I20260619 10:26:19.146840 16205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:19.147053 16205 log.cc:824] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:19.147713 16205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: No bootstrap required, opened a new log
I20260619 10:26:19.148988 16205 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } }
I20260619 10:26:19.149113 16205 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:19.149134 16205 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6ee35d05ec4b0ba455a4eadeefd96a, State: Initialized, Role: FOLLOWER
I20260619 10:26:19.149223 16205 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } }
I20260619 10:26:19.149456 16205 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:19.149454 16212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } } }
I20260619 10:26:19.149564 16212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:26:19.150918 16223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:26:19.150964 16223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:26:19.152338 16137 master_runner.cc:428] Detected that this master ec6ee35d05ec4b0ba455a4eadeefd96a is joining an existing cluster
I20260619 10:26:19.152374 16137 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.90.125:46451
I20260619 10:26:19.158957 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.153320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.9.90.126:33979)
0619 10:26:19.153360 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.153362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.153370 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:19.153423 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.153424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.153434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.153477 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.153859 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.154338 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.155023 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.155225 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.155277 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.156194 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.156196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.156196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.156202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.157632 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.158067 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.158068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.158069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.158135 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.158411 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.158412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.158413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.158550 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.158626 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.158786 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.158863 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:19.158962 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.153168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:47560)
0619 10:26:19.153308 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.153319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.153369 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.153488 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.153489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.153718 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.153778 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.153780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.154404 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.154404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.154957 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.154958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.155299 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.155300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.155363 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.155738 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.155743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.156130 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.157704 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.157705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.157705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.157983 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.158278 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.158279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.158280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.158344 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.158625 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.158626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.158669 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.158754 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.158858 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260619 10:26:19.160189 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.153172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.0.0.1:33968)
0619 10:26:19.153416 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.153427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.153493 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.153621 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.153621 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.153861 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.153939 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.153940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.154821 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.154821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.155646 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.155648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.156093 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.156094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.156157 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.156519 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.156523 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.156869 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.158908 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.158908 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.158909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.159457 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.159645 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.159646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.159646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.159701 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.159928 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.159929 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.159957 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.160004 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.160092 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":79,"threads_started":1}
I20260619 10:26:19.160543 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.153330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.9.90.125:46451)
0619 10:26:19.153357 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.153358 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.153473 (+ 115us) server_negotiation.cc:407] Connection header received
0619 10:26:19.153554 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.153555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.153563 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.153604 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.154021 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.154740 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.155743 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.156020 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.156087 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.156943 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.156944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.156944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.156951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.158844 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.159540 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.159541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.159541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.159591 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.159757 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.159758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.159758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.159855 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.159894 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.160269 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.160348 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.126@KRBTEST.COM for oryx/127.9.90.125@KRBTEST.COM
I20260619 10:26:19.168346 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.161231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.9.90.125:46451)
0619 10:26:19.161264 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.161266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.161449 (+ 183us) server_negotiation.cc:407] Connection header received
0619 10:26:19.161541 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.161542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.161550 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.161590 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.162114 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.162560 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.163646 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.163819 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.163881 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.165919 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.165920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.165921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.165928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.167225 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.167639 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.167640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.167640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.167690 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.167888 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.167889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.167889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.167997 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.168065 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.168232 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.168289 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:26:19.168423 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.161193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.0.0.1:33974)
0619 10:26:19.161340 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.161387 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.161466 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.161598 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.161601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.161982 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.162052 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.162054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.162625 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.162626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.163561 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.163563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.163889 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.163889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.163966 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.164369 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.164373 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.165768 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.167300 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.167307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.167309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.167557 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.167753 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.167754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.167754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.167817 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.168068 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.168073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.168123 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.168206 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.168313 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260619 10:26:19.169034 15673 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.90.125:46451: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a"
member_type: NON_VOTER
last_known_addr {
host: "127.9.90.125"
port: 46451
}
attrs {
promote: true
}
}
dest_uuid: "e9b360c5f4e3437d8dee672b25e2dba4"
cas_config_opid_index: -1
I20260619 10:26:19.169335 15673 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } }
I20260619 10:26:19.169858 16233 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ec6ee35d05ec4b0ba455a4eadeefd96a (127.9.90.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } } }
I20260619 10:26:19.170300 16179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:19.170753 16233 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.90.125:46451
I20260619 10:26:19.170838 16233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } } }
I20260619 10:26:19.170898 16233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:19.170985 16137 master.cc:537] Master@127.9.90.125:46451 shutting down...
I20260619 10:26:19.171175 16179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Refusing update from remote peer e9b360c5f4e3437d8dee672b25e2dba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 10:26:19.171777 16232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 10:26:19.172613 15657 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 10:26:19.173027 15657 consensus_peers.cc:487] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 -> Peer ec6ee35d05ec4b0ba455a4eadeefd96a (127.9.90.125:46451): Couldn't send request to peer ec6ee35d05ec4b0ba455a4eadeefd96a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 10:26:19.173554 16137 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:19.173645 16137 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:19.173697 16137 tablet_replica.cc:330] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: stopping tablet replica
I20260619 10:26:19.186652 16137 master.cc:552] Master@127.9.90.125:46451 shutdown complete.
I20260619 10:26:19.186705 16137 master_runner.cc:305] Clearing existing system tablet
I20260619 10:26:19.186825 16137 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:19.186865 16137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:19.187446 16137 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260619 10:26:19.187891 16237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.188015 16137 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:19.188071 16137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.188093 16137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal
uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "262767d85aabb02759356c38f3c47f52"
server_key_iv: "3aa7b5c2a0a5c1c83e1299c37f3f468a"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.188697 16137 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:19.189821 16137 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 10:26:19.189888 16137 log.cc:1196] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 10:26:19.190052 16137 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Deleting consensus metadata
I20260619 10:26:19.190157 16137 master_runner.cc:315] Copying system tablet from 127.9.90.126:33979
I20260619 10:26:19.190704 16137 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: tablet copy: Beginning tablet copy session from remote peer at address 127.9.90.126:33979
I20260619 10:26:19.196841 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.191062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.9.90.126:33979)
0619 10:26:19.191111 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.191113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.191171 (+ 58us) server_negotiation.cc:407] Connection header received
0619 10:26:19.191246 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.191247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.191255 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.191301 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.191686 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.192222 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.193264 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.193452 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.193498 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.194541 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.194543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.194543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.194549 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.195792 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.196157 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.196157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.196158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.196207 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.196401 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.196402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.196402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.196522 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.196585 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.196713 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.196776 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:26:19.196907 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.190979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:47562)
0619 10:26:19.191104 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.191115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.191187 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.191308 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.191308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.191523 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.191588 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.191590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.192298 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.192298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.193182 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.193184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.193512 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.193513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.193576 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.194081 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.194086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.194458 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.195867 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.195868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.195868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.196093 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.196259 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.196260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.196260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.196319 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.196586 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.196587 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.196632 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.196698 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.196798 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260619 10:26:19.197237 15693 tablet_copy_service.cc:140] P e9b360c5f4e3437d8dee672b25e2dba4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ec6ee35d05ec4b0ba455a4eadeefd96a ({username='oryx', principal='oryx/127.9.90.125@KRBTEST.COM'} at 127.0.0.1:47562)
I20260619 10:26:19.197343 15693 tablet_copy_service.cc:161] P e9b360c5f4e3437d8dee672b25e2dba4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ec6ee35d05ec4b0ba455a4eadeefd96a at {username='oryx', principal='oryx/127.9.90.125@KRBTEST.COM'} at 127.0.0.1:47562: session id = ec6ee35d05ec4b0ba455a4eadeefd96a-00000000000000000000000000000000
I20260619 10:26:19.198220 15693 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 10:26:19.198925 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 10:26:19.199970 16137 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: tablet copy: Starting download of 0 data blocks...
I20260619 10:26:19.200120 16137 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: tablet copy: Starting download of 1 WAL segments...
I20260619 10:26:19.200985 16137 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 10:26:19.201782 15693 tablet_copy_service.cc:342] P e9b360c5f4e3437d8dee672b25e2dba4: Request end of tablet copy session ec6ee35d05ec4b0ba455a4eadeefd96a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.90.125@KRBTEST.COM'} at 127.0.0.1:47562
I20260619 10:26:19.201879 15693 tablet_copy_service.cc:433] P e9b360c5f4e3437d8dee672b25e2dba4: ending tablet copy session ec6ee35d05ec4b0ba455a4eadeefd96a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ec6ee35d05ec4b0ba455a4eadeefd96a
I20260619 10:26:19.204092 16137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:19.205237 16244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.205221 16245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.206648 16137 server_base.cc:1034] running on GCE node
W20260619 10:26:19.206683 16247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.206813 16137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:19.206941 16137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:19.208065 16137 hybrid_clock.cc:648] HybridClock initialized: now 1781864779208057 us; error 21 us; skew 500 ppm
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:19.209316 16137 init.cc:377] Logged in from keytab as oryx/127.9.90.125@KRBTEST.COM (short username oryx)
I20260619 10:26:19.209560 16137 webserver.cc:460] Webserver started at http://127.9.90.125:40127/ using document root <none> and password file <none>
I20260619 10:26:19.209669 16137 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:19.209704 16137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:19.210376 16137 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.210922 16254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.211069 16137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.211117 16137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal
uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "262767d85aabb02759356c38f3c47f52"
server_key_iv: "3aa7b5c2a0a5c1c83e1299c37f3f468a"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.211164 16137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:19.217730 16137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:19.219369 16137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:19.222262 16137 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.125:46451
I20260619 10:26:19.222575 16306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.125:46451 every 8 connection(s)
I20260619 10:26:19.222608 16137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
I20260619 10:26:19.223212 16307 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:26:19.223577 16307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Bootstrap starting.
I20260619 10:26:19.225557 16307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:26:19.225736 16307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Bootstrap complete.
I20260619 10:26:19.225924 16307 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } }
I20260619 10:26:19.225987 16307 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec6ee35d05ec4b0ba455a4eadeefd96a, State: Initialized, Role: LEARNER
I20260619 10:26:19.226022 16307 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } }
I20260619 10:26:19.226202 16307 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:19.226190 16310 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } } }
I20260619 10:26:19.226261 16310 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: This master's current role is: LEARNER
I20260619 10:26:19.227456 16321 catalog_manager.cc:1259] Loaded cluster ID: d1f382c22a204f1b8f634281311475a0
I20260619 10:26:19.227499 16321 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: loading cluster ID for follower catalog manager: success
I20260619 10:26:19.228403 16321 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: acquiring CA information for follower catalog manager: success
I20260619 10:26:19.228653 16321 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:26:19.249969 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.243474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.9.90.125:46451)
0619 10:26:19.243601 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.243602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.243608 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:26:19.243629 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.243630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.243639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.243697 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.244228 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.244959 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.245726 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.245997 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.246053 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.247257 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.247259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.247259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.247267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.248657 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.248996 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.248997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.248997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.249061 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.249236 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.249237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.249237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.249345 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.249437 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.249575 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.249652 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.9.90.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.124.keytab.
Entry for principal oryx/127.9.90.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.124.keytab.
I20260619 10:26:19.274832 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.124:35145
--webserver_interface=127.9.90.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.124.keytab
--principal=oryx/127.9.90.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:33979,127.9.90.125:46451,127.9.90.124:35145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:19.359881 16325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:19.360096 16325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:19.360126 16325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:19.361234 16325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:19.361274 16325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:19.361292 16325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:19.361306 16325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:19.361318 16325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:19.362814 16325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:33979,127.9.90.125:46451,127.9.90.124:35145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.124.keytab
--principal=oryx/127.9.90.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.124:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.90.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:19.363011 16325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:19.363248 16325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:19.367681 16330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.367782 16331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.367681 16333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.367892 16325 server_base.cc:1034] running on GCE node
I20260619 10:26:19.368183 16325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:19.368438 16325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:19.369589 16325 hybrid_clock.cc:648] HybridClock initialized: now 1781864779369566 us; error 41 us; skew 500 ppm
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:19.371866 16325 init.cc:377] Logged in from keytab as oryx/127.9.90.124@KRBTEST.COM (short username oryx)
I20260619 10:26:19.372225 16325 webserver.cc:460] Webserver started at http://127.9.90.124:35067/ using document root <none> and password file <none>
I20260619 10:26:19.372385 16325 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:19.372421 16325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:19.372503 16325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:19.373454 16325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data/instance:
uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "3a0f796155a289a6c06257c2b6f20e99"
server_key_iv: "5dddfae9c95b9f63c985a90eceba6fcf"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.373732 16325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal/instance:
uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "3a0f796155a289a6c06257c2b6f20e99"
server_key_iv: "5dddfae9c95b9f63c985a90eceba6fcf"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.374894 16325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:19.375553 16340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.375702 16325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:19.375774 16325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal
uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "3a0f796155a289a6c06257c2b6f20e99"
server_key_iv: "5dddfae9c95b9f63c985a90eceba6fcf"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.375839 16325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:19.390631 16325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:19.392616 16325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:19.392719 16325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:19.396411 16325 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.124:35145
I20260619 10:26:19.396453 16392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.124:35145 every 8 connection(s)
I20260619 10:26:19.396766 16325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
I20260619 10:26:19.397684 16393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:19.399283 16393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:26:19.399616 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 16325
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.124@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.90.124@KRBTEST.COM
I20260619 10:26:19.409288 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.400282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.9.90.126:33979)
0619 10:26:19.400331 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.400334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.400342 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:19.400800 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.400802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.400812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.400868 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.401527 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.402361 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.403603 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.403915 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.403983 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.406375 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.406377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.406377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.406384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.407765 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.408272 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.408274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.408274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.408355 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.408575 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.408576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.408576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.408703 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.408771 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.409099 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.409196 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:26:19.409402 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.400137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.9.90.124:35145)
0619 10:26:19.400282 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.400286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.400324 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:26:19.400725 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.400739 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.400772 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.400882 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.401528 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.402663 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.403622 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.404077 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.404138 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.406730 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.406744 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.406745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.406750 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.408205 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.408552 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.408554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.408555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.408607 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.408772 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.408772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.408772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.408928 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.409073 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.409188 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.409249 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260619 10:26:19.409402 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.399964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:47568)
0619 10:26:19.400179 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.400226 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.400698 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.400894 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.400897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.401219 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.401438 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.401444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.402466 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.402468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.403486 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.403489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.404003 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.404004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.404107 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.404716 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.404726 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.406294 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.407841 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.407846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.407848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.408190 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.408416 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.408416 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.408417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.408512 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.408767 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.408772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.408920 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.409009 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.409156 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":65,"threads_started":1}
I20260619 10:26:19.410066 16393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.124@KRBTEST.COM for oryx/127.9.90.125@KRBTEST.COM
I20260619 10:26:19.415505 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.410293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.0.0.1:34002)
0619 10:26:19.410341 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.410352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.410392 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.410523 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.410524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.410622 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.410664 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.410665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.411387 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.411388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.411904 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.411906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.412280 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.412281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.412333 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.412628 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.412631 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.413624 (+ 993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.414814 (+ 1190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.414814 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.414815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.414954 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.415119 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.415119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.415119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.415159 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.415312 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.415313 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.415343 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.415386 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.415446 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:26:19.415524 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.410379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.9.90.125:46451)
0619 10:26:19.410439 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.410441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.410447 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:26:19.410470 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.410471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.410481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.410522 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.410741 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.411318 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.411964 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.412224 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.412276 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.413697 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.413698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.413699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.413705 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.414755 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.415004 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.415005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.415005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.415075 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.415191 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.415191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.415192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.415276 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.415316 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.415403 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.415464 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:26:19.415884 16393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.124@KRBTEST.COM for oryx/127.9.90.124@KRBTEST.COM
I20260619 10:26:19.420819 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.416085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.0.0.1:41198)
0619 10:26:19.416124 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.416133 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.416171 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.416256 (+ 85us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.416257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.416350 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.416375 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.416376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.416906 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.416907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.417351 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.417352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.417693 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.417693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.417725 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.418042 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.418047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.419042 (+ 995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.420120 (+ 1078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.420120 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.420121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.420265 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.420394 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.420395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.420395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.420436 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.420585 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.420585 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.420614 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.420656 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.420719 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 10:26:19.420876 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.416127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.9.90.124:35145)
0619 10:26:19.416166 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.416167 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.416175 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:19.416206 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.416207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.416215 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.416252 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.416417 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.416856 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.417429 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.417631 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.417685 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.419109 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.419110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.419110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.419115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.420055 (+ 940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.420314 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.420315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.420315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.420354 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.420469 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.420469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.420469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.420547 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.420587 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.420672 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.420759 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":23}
I20260619 10:26:19.422171 16393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: Bootstrap starting.
I20260619 10:26:19.422940 16393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:19.423182 16393 log.cc:824] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:19.423830 16393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: No bootstrap required, opened a new log
I20260619 10:26:19.425159 16393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } }
I20260619 10:26:19.425284 16393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:19.425307 16393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c3bb4a9f0b445394a1b9a5d39bfcea, State: Initialized, Role: FOLLOWER
I20260619 10:26:19.425426 16393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } }
I20260619 10:26:19.425653 16398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } } }
I20260619 10:26:19.425702 16393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:19.425738 16398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:26:19.427297 16409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:26:19.427349 16409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:26:19.429258 16325 master_runner.cc:428] Detected that this master 69c3bb4a9f0b445394a1b9a5d39bfcea is joining an existing cluster
I20260619 10:26:19.429291 16325 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.90.124:35145
I20260619 10:26:19.436383 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.430517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.9.90.125:46451)
0619 10:26:19.430556 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.430558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.430565 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:19.430584 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.430585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.430602 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.430645 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.431190 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.431918 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.432936 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.433229 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.433265 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.434186 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.434187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.434187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.434194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.435434 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.435720 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.435721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.435721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.435775 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.435926 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.435927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.435927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.436030 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.436082 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.436226 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.436292 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:26:19.436396 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.430350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.0.0.1:34004)
0619 10:26:19.430451 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.430460 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.430507 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.430656 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.430656 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.431026 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.431088 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.431110 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.431997 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.431998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.432851 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.432853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.433348 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.433349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.433400 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.433776 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.433779 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.434103 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.435505 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.435506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.435506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.435663 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.435825 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.435825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.435826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.435879 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.436079 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.436080 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.436141 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.436198 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.436292 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260619 10:26:19.449379 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.430864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.0.0.1:41200)
0619 10:26:19.430979 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.430988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.431048 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.431178 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.431178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.431335 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.431383 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.431384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.432178 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.432179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.432979 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.432981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.433380 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.433380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.433430 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.446300 (+ 12870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.446306 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.446720 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.448323 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.448324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.448324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.448556 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.448734 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.448735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.448735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.448785 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.449048 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.449049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.449089 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.449154 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.449255 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260619 10:26:19.449451 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.430465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.9.90.126:33979)
0619 10:26:19.430517 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.430518 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.430526 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:19.430547 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.430548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.430558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.430603 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.431052 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.431829 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.433048 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.433341 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.433382 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.446915 (+ 13533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.446916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.446917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.446925 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.448380 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.448680 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.448682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.448682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.448749 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.448910 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.448911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.448912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.449029 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.449099 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.449274 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.449356 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:26:19.449432 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.430172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:47584)
0619 10:26:19.430315 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.430326 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.430379 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.430645 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.430646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.430871 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.430959 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.430961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.431922 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.431923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.432949 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.432951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.433455 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.433456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.433516 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.446363 (+ 12847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.446368 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.446822 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.448437 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.448438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.448438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.448624 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.448804 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.448805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.448805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.448862 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.449124 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.449124 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.449159 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.449239 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.449313 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":32,"thread_start_us":72,"threads_started":1}
I20260619 10:26:19.449697 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.430798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41200 (local address 127.9.90.124:35145)
0619 10:26:19.430829 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.430830 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.431032 (+ 202us) server_negotiation.cc:407] Connection header received
0619 10:26:19.431104 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.431105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.431112 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.431135 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.431451 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.432111 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.433045 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.433291 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.433324 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.446800 (+ 13476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.446801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.446802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.446808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.448238 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.448631 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.448631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.448632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.448680 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.448858 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.448859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.448859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.448988 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.449051 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.449184 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.449613 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":341,"server-negotiator.queue_time_us":17}
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864778, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.126@KRBTEST.COM for oryx/127.9.90.124@KRBTEST.COM
I20260619 10:26:19.457408 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.451624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.9.90.124:35145)
0619 10:26:19.451660 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.451662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.451713 (+ 51us) server_negotiation.cc:407] Connection header received
0619 10:26:19.451795 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.451796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.451802 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.451842 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.452121 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.452541 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.453480 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.453648 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.453690 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.455366 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.455367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.455368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.455374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.456483 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.456758 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.456759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.456760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.456808 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.456964 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.456966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.456966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.457063 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.457114 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.457250 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.457316 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:26:19.457412 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.451569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.0.0.1:41214)
0619 10:26:19.451670 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.451680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.451727 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.451855 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.451856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.452009 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.452062 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.452064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.452598 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.452599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.453415 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.453417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.453700 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.453701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.453774 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.454137 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.454142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.455295 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.456552 (+ 1257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.456553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.456554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.456703 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.456858 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.456859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.456859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.456918 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.457119 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.457120 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.457165 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.457223 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.457317 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 10:26:19.457966 15673 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.90.124:35145: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea"
member_type: NON_VOTER
last_known_addr {
host: "127.9.90.124"
port: 35145
}
attrs {
promote: true
}
}
dest_uuid: "e9b360c5f4e3437d8dee672b25e2dba4"
cas_config_opid_index: 5
I20260619 10:26:19.458212 15673 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } }
I20260619 10:26:19.458612 16233 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 69c3bb4a9f0b445394a1b9a5d39bfcea (127.9.90.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.458716 16367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:19.459312 16233 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.90.124:35145
I20260619 10:26:19.459390 16233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.459448 16233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:19.459522 16325 master.cc:537] Master@127.9.90.124:35145 shutting down...
I20260619 10:26:19.459615 16367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 FOLLOWER]: Refusing update from remote peer e9b360c5f4e3437d8dee672b25e2dba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 10:26:19.459904 16232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 10:26:19.461608 15660 consensus_peers.cc:487] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 -> Peer 69c3bb4a9f0b445394a1b9a5d39bfcea (127.9.90.124:35145): Couldn't send request to peer 69c3bb4a9f0b445394a1b9a5d39bfcea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 10:26:19.463205 16325 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:19.463294 16325 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:19.463340 16325 tablet_replica.cc:330] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: stopping tablet replica
I20260619 10:26:19.464536 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.459525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.9.90.125:46451)
0619 10:26:19.459597 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.459598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.460642 (+ 1044us) server_negotiation.cc:407] Connection header received
0619 10:26:19.460671 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.460672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.460696 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.460748 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:19.460901 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.461589 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.463104 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.463702 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.463784 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.463858 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.464429 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.464472 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 10:26:19.464960 16344 connection.cc:441] server connection from 127.0.0.1:41188 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:41188 (request call id 30) could send its response
I20260619 10:26:19.465046 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.458570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.0.0.1:34012)
0619 10:26:19.460543 (+ 1973us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.460554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.460594 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.460800 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.460801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.460802 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:19.460839 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.460840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.461669 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.461670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.462921 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.462924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.463891 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.463892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.463959 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.463980 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.464027 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.464062 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1957}
I20260619 10:26:19.465346 16281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 LEARNER]: Refusing update from remote peer e9b360c5f4e3437d8dee672b25e2dba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 10:26:19.465561 16232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 10:26:19.466274 16281 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 69c3bb4a9f0b445394a1b9a5d39bfcea (127.9.90.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.466403 16325 master.cc:552] Master@127.9.90.124:35145 shutdown complete.
I20260619 10:26:19.466434 16325 master_runner.cc:305] Clearing existing system tablet
I20260619 10:26:19.466534 16325 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:19.466568 16325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:19.467108 16325 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:19.467314 16310 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b360c5f4e3437d8dee672b25e2dba4. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.467387 16310 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: This master's current role is: LEARNER
I20260619 10:26:19.467415 16310 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: NON_VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: true } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.467453 16310 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: This master's current role is: LEARNER
I20260619 10:26:19.467475 16421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.467657 16232 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: attempting to promote NON_VOTER ec6ee35d05ec4b0ba455a4eadeefd96a to VOTER
I20260619 10:26:19.467828 16232 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } }
I20260619 10:26:19.468230 16281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 LEARNER]: Refusing update from remote peer e9b360c5f4e3437d8dee672b25e2dba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 10:26:19.468276 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.468176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.0.0.1:41222)
0619 10:26:19.468213 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.468250 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 10:26:19.468295 16325 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:19.468339 16325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:26:19.468361 16325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal
uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "3a0f796155a289a6c06257c2b6f20e99"
server_key_iv: "5dddfae9c95b9f63c985a90eceba6fcf"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.468513 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.468452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.0.0.1:41238)
0619 10:26:19.468478 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.468496 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 10:26:19.468590 16422 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 10:26:19.468743 15660 consensus_peers.cc:487] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 -> Peer 69c3bb4a9f0b445394a1b9a5d39bfcea (127.9.90.124:35145): Couldn't send request to peer 69c3bb4a9f0b445394a1b9a5d39bfcea. Status: Network error: Client connection negotiation failed: client connection to 127.9.90.124:35145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 10:26:19.469105 16325 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:19.469198 16233 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ec6ee35d05ec4b0ba455a4eadeefd96a (127.9.90.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.470117 16233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.470193 16233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:19.470235 16233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.470280 16233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:19.470404 16325 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 10:26:19.470495 16325 log.cc:1196] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 10:26:19.470563 16281 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ec6ee35d05ec4b0ba455a4eadeefd96a (127.9.90.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.470721 16325 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: Deleting consensus metadata
I20260619 10:26:19.470824 16325 master_runner.cc:315] Copying system tablet from 127.9.90.126:33979
I20260619 10:26:19.471382 16325 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: tablet copy: Beginning tablet copy session from remote peer at address 127.9.90.126:33979
I20260619 10:26:19.471395 16420 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.471465 16420 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:26:19.477396 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.471623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:47598)
0619 10:26:19.471745 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.471754 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.471834 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.471971 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.471971 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.472112 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.472156 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.472157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.473048 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.473048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.473624 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.473626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.474103 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.474104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.474153 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.474538 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.474543 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.474855 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.476485 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.476486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.476486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.476672 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.476862 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.476863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.476863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.476910 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.477140 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.477140 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.477177 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.477224 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.477302 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":87,"threads_started":1}
I20260619 10:26:19.477418 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.471660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.9.90.126:33979)
0619 10:26:19.471697 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.471699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.471804 (+ 105us) server_negotiation.cc:407] Connection header received
0619 10:26:19.471896 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.471897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.471907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.471968 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.472208 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.472958 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.473694 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.474035 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.474103 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.474928 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.474930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.474930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.474936 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.476399 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.476744 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.476745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.476745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.476806 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.476957 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.476958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.476958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.477084 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.477151 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.477252 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.477327 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:26:19.477586 15693 tablet_copy_service.cc:140] P e9b360c5f4e3437d8dee672b25e2dba4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 69c3bb4a9f0b445394a1b9a5d39bfcea ({username='oryx', principal='oryx/127.9.90.124@KRBTEST.COM'} at 127.0.0.1:47598)
I20260619 10:26:19.477623 15693 tablet_copy_service.cc:161] P e9b360c5f4e3437d8dee672b25e2dba4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 69c3bb4a9f0b445394a1b9a5d39bfcea at {username='oryx', principal='oryx/127.9.90.124@KRBTEST.COM'} at 127.0.0.1:47598: session id = 69c3bb4a9f0b445394a1b9a5d39bfcea-00000000000000000000000000000000
I20260619 10:26:19.478253 15693 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 10:26:19.478703 16325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 10:26:19.479658 16325 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: tablet copy: Starting download of 0 data blocks...
I20260619 10:26:19.479784 16325 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: tablet copy: Starting download of 1 WAL segments...
I20260619 10:26:19.480391 16325 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 10:26:19.481084 15693 tablet_copy_service.cc:342] P e9b360c5f4e3437d8dee672b25e2dba4: Request end of tablet copy session 69c3bb4a9f0b445394a1b9a5d39bfcea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.90.124@KRBTEST.COM'} at 127.0.0.1:47598
I20260619 10:26:19.481140 15693 tablet_copy_service.cc:433] P e9b360c5f4e3437d8dee672b25e2dba4: ending tablet copy session 69c3bb4a9f0b445394a1b9a5d39bfcea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 69c3bb4a9f0b445394a1b9a5d39bfcea
I20260619 10:26:19.483291 16325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:19.484220 16431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.484341 16432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.484479 16325 server_base.cc:1034] running on GCE node
W20260619 10:26:19.490531 16434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.490710 16325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:19.490852 16325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:19.491981 16325 hybrid_clock.cc:648] HybridClock initialized: now 1781864779491974 us; error 36 us; skew 500 ppm
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:19.493243 16325 init.cc:377] Logged in from keytab as oryx/127.9.90.124@KRBTEST.COM (short username oryx)
I20260619 10:26:19.493518 16325 webserver.cc:460] Webserver started at http://127.9.90.124:41495/ using document root <none> and password file <none>
I20260619 10:26:19.493637 16325 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:19.493672 16325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:19.494282 16325 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.494763 16441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.494884 16325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.494925 16325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal
uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea"
format_stamp: "Formatted at 2026-06-19 10:26:19 on dist-test-slave-hw7q"
server_key: "3a0f796155a289a6c06257c2b6f20e99"
server_key_iv: "5dddfae9c95b9f63c985a90eceba6fcf"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.494980 16325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:19.506193 16325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:19.507658 16325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:19.510428 16325 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.124:35145
I20260619 10:26:19.510648 16493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.124:35145 every 8 connection(s)
I20260619 10:26:19.510788 16325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
I20260619 10:26:19.511461 16494 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:26:19.511850 16494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: Bootstrap starting.
I20260619 10:26:19.514103 16494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:26:19.514302 16494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: Bootstrap complete.
I20260619 10:26:19.514467 16494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } }
I20260619 10:26:19.514534 16494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69c3bb4a9f0b445394a1b9a5d39bfcea, State: Initialized, Role: LEARNER
I20260619 10:26:19.514559 16494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } }
I20260619 10:26:19.514703 16494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:19.514729 16497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.514858 16497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: This master's current role is: LEARNER
I20260619 10:26:19.516479 16509 catalog_manager.cc:1259] Loaded cluster ID: d1f382c22a204f1b8f634281311475a0
I20260619 10:26:19.516557 16509 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: loading cluster ID for follower catalog manager: success
I20260619 10:26:19.518184 16509 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: acquiring CA information for follower catalog manager: success
I20260619 10:26:19.518514 16509 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:26:19.522344 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.515557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.9.90.124:35145)
0619 10:26:19.515655 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.515657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.515704 (+ 47us) server_negotiation.cc:407] Connection header received
0619 10:26:19.515768 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.515769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.515780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.516628 (+ 848us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.517102 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.517890 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.518638 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.518841 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.518896 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.520014 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.520016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.520016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.520023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.521321 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.521638 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.521639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.521639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.521688 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.521875 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.521876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.521877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.521978 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.522061 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.522164 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.522250 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":73,"thread_start_us":57,"threads_started":1}
I20260619 10:26:19.522797 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 15735
I20260619 10:26:19.527405 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:37485
--local_ip_for_outbound_sockets=127.9.90.65
--tserver_master_addrs=127.9.90.126:33979
--webserver_port=42787
--webserver_interface=127.9.90.65
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab
--principal=oryx/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.90.126:33979,127.9.90.125:46451
--tserver_master_addrs=127.9.90.126:33979,127.9.90.125:46451,127.9.90.124:35145 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:19.606331 16510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:19.606542 16510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:19.606572 16510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:19.607712 16510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:19.607755 16510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:19.607790 16510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:19.609227 16510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab
--principal=oryx/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:37485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=42787
--tserver_master_addrs=127.9.90.126:33979,127.9.90.125:46451,127.9.90.124:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:19.609457 16510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:19.609675 16510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:19.614521 16515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.614492 16518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.614487 16516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.614972 16510 server_base.cc:1034] running on GCE node
I20260619 10:26:19.615163 16510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:19.615423 16510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:19.616564 16510 hybrid_clock.cc:648] HybridClock initialized: now 1781864779616552 us; error 27 us; skew 500 ppm
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:19.618852 16510 init.cc:377] Logged in from keytab as oryx/127.9.90.65@KRBTEST.COM (short username oryx)
I20260619 10:26:19.619251 16510 webserver.cc:460] Webserver started at http://127.9.90.65:42787/ using document root <none> and password file <none>
I20260619 10:26:19.619436 16510 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:19.619488 16510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:19.620710 16510 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:19.621198 16525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.621346 16510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.621424 16510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "83c8715c390f407da0848f3d08206803"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "b98ffa148769409d7a0c6d35b8a449fd"
server_key_iv: "9bc41f1e790ab15704f43eab8619b366"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.621737 16510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:19.640925 16510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:19.643029 16510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:19.643162 16510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:19.643378 16510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:19.643653 16510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:19.643685 16510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.643712 16510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:19.643733 16510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.649240 16510 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:37485
I20260619 10:26:19.649302 16638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:37485 every 8 connection(s)
I20260619 10:26:19.649598 16510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:19.651700 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 16510
I20260619 10:26:19.651847 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 15869
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.125@KRBTEST.COM
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.124@KRBTEST.COM
I20260619 10:26:19.658365 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:45495
--local_ip_for_outbound_sockets=127.9.90.66
--tserver_master_addrs=127.9.90.126:33979
--webserver_port=46403
--webserver_interface=127.9.90.66
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab
--principal=oryx/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.90.126:33979,127.9.90.125:46451
--tserver_master_addrs=127.9.90.126:33979,127.9.90.125:46451,127.9.90.124:35145 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:26:19.660746 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.650369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:43303 (local address 127.9.90.126:33979)
0619 10:26:19.650467 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.650469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.650704 (+ 235us) server_negotiation.cc:407] Connection header received
0619 10:26:19.651163 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.651164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.651176 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.651229 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.652419 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.653134 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.654552 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.654854 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.654893 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.657536 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.657538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.657538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.657546 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.659257 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.659746 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.659747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.659748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.659812 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.660058 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.660059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.660059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.660181 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.660220 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.660569 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.660648 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 10:26:19.661106 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.650570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:44821 (local address 127.9.90.125:46451)
0619 10:26:19.650610 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.650611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.650644 (+ 33us) server_negotiation.cc:407] Connection header received
0619 10:26:19.651192 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.651194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.651204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.651234 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.652217 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.653001 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.654555 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.654860 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.654901 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.658490 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.658491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.658492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.658499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.659959 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.660335 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.660336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.660336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.660392 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.660583 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.660583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.660584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.660715 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.660755 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.660946 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.661020 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:26:19.661619 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.650418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:32853 (local address 127.9.90.124:35145)
0619 10:26:19.650505 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.650506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.650667 (+ 161us) server_negotiation.cc:407] Connection header received
0619 10:26:19.651192 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.651193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.651204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.651234 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.652217 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.652988 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.654554 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.654866 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.654901 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.658921 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.658923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.658923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.658932 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.660546 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.660886 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.660887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.660888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.660944 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.661123 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.661124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.661124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.661241 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.661283 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.661449 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.661540 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 10:26:19.661645 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.650273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.9.90.65:44821)
0619 10:26:19.650525 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.650566 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.651085 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.651324 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.651329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.651811 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.652109 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.652115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.653122 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.653123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.654455 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.654458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.654976 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.654977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.655107 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.655725 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.655734 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.658393 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.660043 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.660044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.660045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.660233 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.660462 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.660462 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.660463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.660516 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.660801 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.660801 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.660840 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.661192 (+ 352us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.661443 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":220,"thread_start_us":71,"threads_started":1}
I20260619 10:26:19.661648 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.650270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.65:43303)
0619 10:26:19.650649 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.650659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.651084 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.651583 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.651584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.651836 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.652326 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.652328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.653222 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.653223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.654456 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.654458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.654974 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.654975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.655107 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.655895 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.655901 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.657432 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.659343 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.659350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.659353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.659647 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.659875 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.659876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.659877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.659997 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.660261 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.660266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.660421 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.661024 (+ 603us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.661372 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":536,"thread_start_us":70,"threads_started":1}
I20260619 10:26:19.662175 16641 heartbeater.cc:347] Connected to a master server at 127.9.90.125:46451
I20260619 10:26:19.662222 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.650270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.9.90.65:32853)
0619 10:26:19.650610 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.650618 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.651078 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.651324 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.651329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.651811 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.652109 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.652115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.653122 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.653123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.654455 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.654458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.654978 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.654979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.655107 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.655788 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.655792 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.658626 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.660621 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.660622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.660622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.660812 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.661009 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.661009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.661010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.661060 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.661322 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.661322 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.661351 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.661892 (+ 541us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.661982 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":125,"thread_start_us":80,"threads_started":1}
I20260619 10:26:19.662277 16641 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:19.662420 16640 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33979
I20260619 10:26:19.662461 16640 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:19.662520 16639 heartbeater.cc:347] Connected to a master server at 127.9.90.124:35145
I20260619 10:26:19.662568 16639 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:19.662618 16639 heartbeater.cc:511] Master 127.9.90.124:35145 requested a full tablet report, sending...
I20260619 10:26:19.662529 16640 heartbeater.cc:511] Master 127.9.90.126:33979 requested a full tablet report, sending...
I20260619 10:26:19.662462 16641 heartbeater.cc:511] Master 127.9.90.125:46451 requested a full tablet report, sending...
I20260619 10:26:19.663091 15673 ts_manager.cc:194] Re-registered known tserver with Master: 83c8715c390f407da0848f3d08206803 (127.9.90.65:37485)
I20260619 10:26:19.663127 16458 ts_manager.cc:194] Registered new tserver with Master: 83c8715c390f407da0848f3d08206803 (127.9.90.65:37485)
I20260619 10:26:19.663566 15673 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:43303
I20260619 10:26:19.663987 16271 ts_manager.cc:194] Registered new tserver with Master: 83c8715c390f407da0848f3d08206803 (127.9.90.65:37485)
W20260619 10:26:19.738538 16646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:19.738729 16646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:19.738760 16646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:19.739902 16646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:19.739943 16646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:19.739977 16646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:19.741430 16646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab
--principal=oryx/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:45495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=46403
--tserver_master_addrs=127.9.90.126:33979,127.9.90.125:46451,127.9.90.124:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:19.741653 16646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:19.741906 16646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:19.746439 16651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.746469 16654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.746443 16652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.746459 16646 server_base.cc:1034] running on GCE node
I20260619 10:26:19.746779 16646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:19.747035 16646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:19.748205 16646 hybrid_clock.cc:648] HybridClock initialized: now 1781864779748189 us; error 32 us; skew 500 ppm
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:19.750579 16646 init.cc:377] Logged in from keytab as oryx/127.9.90.66@KRBTEST.COM (short username oryx)
I20260619 10:26:19.751001 16646 webserver.cc:460] Webserver started at http://127.9.90.66:46403/ using document root <none> and password file <none>
I20260619 10:26:19.751214 16646 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:19.751273 16646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:19.752570 16646 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:19.753103 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.753257 16646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:19.753330 16646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "ad5e370ed46448198c12796120515e87"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "7ad18588a29112efde2fd07cb2784ff1"
server_key_iv: "3940355451a07384e69a8913d127cab2"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.753641 16646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:19.780515 16646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:19.782284 16646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:19.782408 16646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:19.782617 16646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:19.782908 16646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:19.782941 16646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.782970 16646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:19.782994 16646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.788305 16646 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:45495
I20260619 10:26:19.788362 16774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:45495 every 8 connection(s)
I20260619 10:26:19.788657 16646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:19.793501 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 16646
I20260619 10:26:19.793651 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 16003
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.125@KRBTEST.COM
I20260619 10:26:19.798803 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.789678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:55751 (local address 127.9.90.125:46451)
0619 10:26:19.789719 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.789721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.789729 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:19.790260 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.790261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.790273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.790318 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.791002 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.791761 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.792595 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.792853 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.792915 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.796227 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.796228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.796229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.796236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.797491 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.797955 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.797956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.797956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.798010 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.798176 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.798177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.798178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.798330 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.798386 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.798630 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.798706 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:19.799031 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:45345
--local_ip_for_outbound_sockets=127.9.90.67
--tserver_master_addrs=127.9.90.126:33979
--webserver_port=33649
--webserver_interface=127.9.90.67
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab
--principal=oryx/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.90.126:33979,127.9.90.125:46451
--tserver_master_addrs=127.9.90.126:33979,127.9.90.125:46451,127.9.90.124:35145 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:26:19.799167 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.789351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.66:58037)
0619 10:26:19.789876 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.789883 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.790169 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.790735 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.790736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.790870 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.790924 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.790930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.791597 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.791598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.792628 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.792630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.793053 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.793054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.793117 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.793544 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.793554 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.795701 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.797385 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.797392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.797394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.797652 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.797869 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.797869 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.797870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.798014 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.798325 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.798332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.798488 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.798591 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.799072 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":365,"thread_start_us":80,"threads_started":1}
I20260619 10:26:19.799214 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.789659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:58037 (local address 127.9.90.126:33979)
0619 10:26:19.789711 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.789714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.789928 (+ 214us) server_negotiation.cc:407] Connection header received
0619 10:26:19.790287 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.790288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.790299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.790327 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.791023 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.791506 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.792717 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.792990 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.793043 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.795825 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.795826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.795826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.795834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.797301 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.797719 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.797720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.797720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.797783 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.798100 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.798101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.798101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.798265 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.798325 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.799066 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.799139 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:26:19.799248 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.789351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.9.90.66:55751)
0619 10:26:19.789537 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.789578 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.790161 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.790403 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.790407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.790730 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.790923 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.790927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.791868 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.791869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.792518 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.792520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.792923 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.792923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.793069 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.793662 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.793667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.796125 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.797640 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.797640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.797641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.797848 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.798063 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.798064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.798064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.798118 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.798387 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.798388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.798488 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.798589 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.798774 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":22,"thread_start_us":96,"threads_started":1}
I20260619 10:26:19.799607 16776 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33979
I20260619 10:26:19.799615 16777 heartbeater.cc:347] Connected to a master server at 127.9.90.125:46451
I20260619 10:26:19.799712 16776 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:19.799712 16777 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:19.799904 16776 heartbeater.cc:511] Master 127.9.90.126:33979 requested a full tablet report, sending...
I20260619 10:26:19.799904 16777 heartbeater.cc:511] Master 127.9.90.125:46451 requested a full tablet report, sending...
I20260619 10:26:19.800307 15673 ts_manager.cc:194] Re-registered known tserver with Master: ad5e370ed46448198c12796120515e87 (127.9.90.66:45495)
I20260619 10:26:19.800349 16271 ts_manager.cc:194] Registered new tserver with Master: ad5e370ed46448198c12796120515e87 (127.9.90.66:45495)
I20260619 10:26:19.800729 15673 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:58037
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.124@KRBTEST.COM
I20260619 10:26:19.804067 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.789351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.9.90.66:50265)
0619 10:26:19.789753 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.789762 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.790168 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.790404 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.790408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.790773 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.790924 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.790930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.791839 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.791840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.792720 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.792722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.793122 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.793123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.793174 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.800350 (+ 7176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.800355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.801608 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.803190 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.803191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.803191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.803356 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.803540 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.803540 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.803540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.803585 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.803831 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.803831 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.803863 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.803909 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.803985 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":72,"thread_start_us":95,"threads_started":1}
I20260619 10:26:19.804100 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.789637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:50265 (local address 127.9.90.124:35145)
0619 10:26:19.789691 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.789693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.789828 (+ 135us) server_negotiation.cc:407] Connection header received
0619 10:26:19.790263 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.790264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.790279 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.790311 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.791033 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.791764 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.792782 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.793014 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.793072 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.801684 (+ 8612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.801686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.801686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.801695 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.803118 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.803421 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.803422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.803423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.803481 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.803639 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.803640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.803640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.803774 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.803842 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.803935 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.804011 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:26:19.804418 16775 heartbeater.cc:347] Connected to a master server at 127.9.90.124:35145
I20260619 10:26:19.804469 16775 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:19.804541 16775 heartbeater.cc:511] Master 127.9.90.124:35145 requested a full tablet report, sending...
I20260619 10:26:19.804848 16458 ts_manager.cc:194] Registered new tserver with Master: ad5e370ed46448198c12796120515e87 (127.9.90.66:45495)
W20260619 10:26:19.879793 16782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:19.879982 16782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:19.880003 16782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:19.881122 16782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:19.881160 16782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:19.881186 16782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:19.882786 16782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab
--principal=oryx/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:45345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=33649
--tserver_master_addrs=127.9.90.126:33979,127.9.90.125:46451,127.9.90.124:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:19.882970 16782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:19.883140 16782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:19.887741 16790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:19.887800 16788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.887833 16782 server_base.cc:1034] running on GCE node
W20260619 10:26:19.887773 16787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:19.888084 16782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:19.888283 16782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:19.889432 16782 hybrid_clock.cc:648] HybridClock initialized: now 1781864779889426 us; error 29 us; skew 500 ppm
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:19.891669 16782 init.cc:377] Logged in from keytab as oryx/127.9.90.67@KRBTEST.COM (short username oryx)
I20260619 10:26:19.892086 16782 webserver.cc:460] Webserver started at http://127.9.90.67:33649/ using document root <none> and password file <none>
I20260619 10:26:19.892278 16782 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:19.892338 16782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:19.893631 16782 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:19.894253 16797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.894388 16782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.894469 16782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "ad4bf6b0bc984d66856c3ad71c7ead83"
format_stamp: "Formatted at 2026-06-19 10:26:18 on dist-test-slave-hw7q"
server_key: "6b099c1d25385aa99dcacb548fad407f"
server_key_iv: "05048ee8612b59655c982a5f88e8fbe1"
server_key_version: "encryptionkey@0"
I20260619 10:26:19.894788 16782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:19.905205 16782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:19.906801 16782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:19.906926 16782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:19.907109 16782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:19.907378 16782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:19.907409 16782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.907437 16782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:19.907460 16782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:19.913028 16782 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:45345
I20260619 10:26:19.913077 16910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:45345 every 8 connection(s)
I20260619 10:26:19.913401 16782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:26:19.913899 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 16782
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.125@KRBTEST.COM
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:19.923342 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.915258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.9.90.124:35145)
0619 10:26:19.915386 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.915388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.915593 (+ 205us) server_negotiation.cc:407] Connection header received
0619 10:26:19.915728 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.915729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.915741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.915781 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.916605 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.917303 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.918371 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.918650 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.918686 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.920192 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.920193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.920194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.920200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.921872 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.922216 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.922217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.922217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.922302 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.922504 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.922505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.922506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.922625 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.922684 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.923159 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.923235 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
Jun 19 10:26:19 dist-test-slave-hw7q krb5kdc[15628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.124@KRBTEST.COM
I20260619 10:26:19.927148 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.915031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:44623 (local address 127.9.90.124:35145)
0619 10:26:19.915078 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.915080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.917561 (+ 2481us) server_negotiation.cc:407] Connection header received
0619 10:26:19.919498 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.919499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.919522 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.919557 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.920359 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.920994 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.922015 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.922288 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.922344 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.924425 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.924426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.924426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.924432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.925878 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.926279 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.926279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.926280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.926351 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.926578 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.926579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.926579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.926695 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.926757 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.926968 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.927035 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:26:19.927182 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.914752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:37607 (local address 127.9.90.126:33979)
0619 10:26:19.914808 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.914810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.914820 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:19.915097 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.915099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.915113 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.915146 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.916072 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.916756 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.918275 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.918551 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.918591 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.923445 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.923446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.923447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.923454 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.925458 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.925944 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.925945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.925945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.926031 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.926384 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.926385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.926385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.926536 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.926597 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.926975 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.927074 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:26:19.927544 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.914530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.67:37607)
0619 10:26:19.914691 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.914701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.914964 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.915277 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.915281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.915749 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.915930 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.915936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.916877 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.916878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.918172 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.918175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.918671 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.918672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.918801 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.919486 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.919491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.922899 (+ 3408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.925526 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.925527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.925528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.925848 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.926138 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.926139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.926139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.926275 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.926648 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.926655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.926812 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.927023 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.927299 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":54,"thread_start_us":72,"threads_started":1}
I20260619 10:26:19.927804 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.914970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.9.90.67:44623)
0619 10:26:19.917477 (+ 2507us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.917488 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.918530 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.919623 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.919624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.920192 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.920262 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.920264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.921100 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.921100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.921892 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.921895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.922390 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.922391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.922456 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.922866 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.922871 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.924344 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.925963 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.925963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.925964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.926190 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.926439 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.926440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.926440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.926499 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.926756 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.926756 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.926823 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.927407 (+ 584us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.927552 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2488,"thread_start_us":65,"threads_started":1}
I20260619 10:26:19.928129 16912 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33979
I20260619 10:26:19.928164 16911 heartbeater.cc:347] Connected to a master server at 127.9.90.124:35145
I20260619 10:26:19.928218 16912 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:19.928422 16912 heartbeater.cc:511] Master 127.9.90.126:33979 requested a full tablet report, sending...
I20260619 10:26:19.928762 15673 ts_manager.cc:194] Re-registered known tserver with Master: ad4bf6b0bc984d66856c3ad71c7ead83 (127.9.90.67:45345)
I20260619 10:26:19.929049 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.914410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:53619 (local address 127.9.90.125:46451)
0619 10:26:19.914684 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.914687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.914699 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:26:19.915068 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.915070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.915084 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.915112 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.916022 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.916765 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.918264 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.918537 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.918582 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.921705 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.921706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.921707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.921713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.923337 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.927746 (+ 4409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.927747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.927747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.927814 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.928145 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.928146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.928147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.928289 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.928336 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.928888 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.928963 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260619 10:26:19.929185 15673 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:37607
I20260619 10:26:19.929234 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.914103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.9.90.67:53619)
0619 10:26:19.914271 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.914307 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.914949 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.915277 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.915281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.915647 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:19.915930 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.915936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.916880 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.916881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.918171 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.918173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.918665 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.918666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.918801 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.919301 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:19.919311 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:19.921596 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:19.923692 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.923698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.923700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.927652 (+ 3952us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.927974 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:19.927975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:19.927975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:19.928066 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:19.928397 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:19.928398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:19.928439 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.928845 (+ 406us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.928945 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":54,"thread_start_us":98,"threads_started":1}
I20260619 10:26:19.928220 16911 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:19.929848 16911 heartbeater.cc:511] Master 127.9.90.124:35145 requested a full tablet report, sending...
I20260619 10:26:19.929981 16913 heartbeater.cc:347] Connected to a master server at 127.9.90.125:46451
I20260619 10:26:19.930017 16913 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:19.930073 16913 heartbeater.cc:511] Master 127.9.90.125:46451 requested a full tablet report, sending...
I20260619 10:26:19.930146 16458 ts_manager.cc:194] Registered new tserver with Master: ad4bf6b0bc984d66856c3ad71c7ead83 (127.9.90.67:45345)
I20260619 10:26:19.930250 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.915680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.9.90.126:33979)
0619 10:26:19.916576 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.916578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.916735 (+ 157us) server_negotiation.cc:407] Connection header received
0619 10:26:19.916755 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.916756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.916767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.916829 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.917724 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.918439 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.920252 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.920803 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.920866 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.924355 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.924356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.924356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.924363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.927417 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.928192 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.928193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.928194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.928251 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.928623 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.928624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.928624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.929324 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.929410 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.930083 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.930163 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":86,"threads_started":1}
I20260619 10:26:19.930336 16271 ts_manager.cc:194] Registered new tserver with Master: ad4bf6b0bc984d66856c3ad71c7ead83 (127.9.90.67:45345)
I20260619 10:26:19.931983 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.916304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.9.90.125:46451)
0619 10:26:19.920941 (+ 4637us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.920942 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.923755 (+ 2813us) server_negotiation.cc:407] Connection header received
0619 10:26:19.923775 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.923776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.923787 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.923837 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:19.924375 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.925056 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.925985 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.926264 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.926315 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.927844 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:19.927845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:19.927846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:19.927853 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:19.930929 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.931285 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.931286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.931286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.931352 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:19.931575 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:19.931576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:19.931576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:19.931675 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:19.931752 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.931866 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.931926 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4619,"thread_start_us":49,"threads_started":1}
I20260619 10:26:19.931764 15673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:19.933382 15673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:19.939983 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.936500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:37485 (local address 127.0.0.1:52330)
0619 10:26:19.936676 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.936686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.936721 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.937184 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.937184 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.937185 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:19.937218 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.937220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.938215 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.938215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.938889 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.938891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.939757 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.939758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.939815 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.939820 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.939880 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.939918 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":2}
I20260619 10:26:19.940006 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.936638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.9.90.65:37485)
0619 10:26:19.937030 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.937033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.937041 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:19.937074 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.937076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.937092 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.937294 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:19.937348 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.938133 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.938961 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.939670 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.939745 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.939807 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.939894 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.939929 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":68,"threads_started":1}
I20260619 10:26:19.940333 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.936561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45495 (local address 127.0.0.1:45858)
0619 10:26:19.936846 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.936856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.936899 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.937170 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.937171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.937173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:19.937218 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.937220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.938049 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.938050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.939339 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.939342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.940123 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.940124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.940171 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.940174 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.940217 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.940262 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260619 10:26:19.940358 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.936887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.9.90.66:45495)
0619 10:26:19.937025 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.937029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.937040 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:26:19.937079 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.937081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.937096 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.937168 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:19.937275 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.937968 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.939406 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.939960 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.940014 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.940091 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.940245 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.940283 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260619 10:26:19.941075 16709 tablet_service.cc:1467] Processing CreateTablet for tablet 4abf63ed99ee4a95874e58d1924996a4 (DEFAULT_TABLE table=test-table [id=57b34c1b96534c45be523a35e669ba23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:19.941115 16573 tablet_service.cc:1467] Processing CreateTablet for tablet 4abf63ed99ee4a95874e58d1924996a4 (DEFAULT_TABLE table=test-table [id=57b34c1b96534c45be523a35e669ba23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:19.941425 16709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abf63ed99ee4a95874e58d1924996a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:19.941468 16573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abf63ed99ee4a95874e58d1924996a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:19.941574 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.936564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:45345 (local address 127.0.0.1:39862)
0619 10:26:19.936883 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.936890 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.936927 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.938090 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.938091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.938093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:19.938134 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.938136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.939278 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.939279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.940315 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.940318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.941312 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.941313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.941429 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.941433 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.941474 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.941511 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260619 10:26:19.941910 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.936763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.9.90.67:45345)
0619 10:26:19.937903 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.937907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.937916 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:26:19.937946 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.937948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.937970 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.938039 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:19.938203 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.939207 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.940479 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.941172 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.941651 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.941727 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.941775 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.941842 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1110,"thread_start_us":52,"threads_started":1}
I20260619 10:26:19.942471 16845 tablet_service.cc:1467] Processing CreateTablet for tablet 4abf63ed99ee4a95874e58d1924996a4 (DEFAULT_TABLE table=test-table [id=57b34c1b96534c45be523a35e669ba23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:19.942754 16845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abf63ed99ee4a95874e58d1924996a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:19.944676 16938 tablet_bootstrap.cc:492] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: Bootstrap starting.
I20260619 10:26:19.945168 16938 tablet_bootstrap.cc:654] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:19.945192 16937 tablet_bootstrap.cc:492] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: Bootstrap starting.
I20260619 10:26:19.945439 16938 log.cc:824] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:19.946000 16937 tablet_bootstrap.cc:654] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:19.946178 16938 tablet_bootstrap.cc:492] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: No bootstrap required, opened a new log
I20260619 10:26:19.946249 16938 ts_tablet_manager.cc:1397] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:19.946386 16937 log.cc:824] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:19.946725 16940 tablet_bootstrap.cc:492] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: Bootstrap starting.
I20260619 10:26:19.947345 16940 tablet_bootstrap.cc:654] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:19.947642 16937 tablet_bootstrap.cc:492] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: No bootstrap required, opened a new log
I20260619 10:26:19.947698 16937 ts_tablet_manager.cc:1397] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 10:26:19.947933 16938 raft_consensus.cc:348] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:19.948108 16938 raft_consensus.cc:374] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:19.948129 16938 raft_consensus.cc:729] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c8715c390f407da0848f3d08206803, State: Initialized, Role: FOLLOWER
I20260619 10:26:19.948240 16938 consensus_queue.cc:260] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:19.948495 16640 heartbeater.cc:503] Master 127.9.90.126:33979 was elected leader, sending a full tablet report...
I20260619 10:26:19.949041 16938 ts_tablet_manager.cc:1428] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:26:19.950152 16940 log.cc:824] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:19.950080 16937 raft_consensus.cc:348] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:19.950222 16937 raft_consensus.cc:374] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:19.950273 16937 raft_consensus.cc:729] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad5e370ed46448198c12796120515e87, State: Initialized, Role: FOLLOWER
I20260619 10:26:19.950412 16937 consensus_queue.cc:260] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:19.950624 16937 ts_tablet_manager.cc:1428] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:26:19.950702 16776 heartbeater.cc:503] Master 127.9.90.126:33979 was elected leader, sending a full tablet report...
I20260619 10:26:19.950850 16940 tablet_bootstrap.cc:492] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: No bootstrap required, opened a new log
I20260619 10:26:19.950914 16940 ts_tablet_manager.cc:1397] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 10:26:19.952848 16940 raft_consensus.cc:348] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:19.952970 16940 raft_consensus.cc:374] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:19.952999 16940 raft_consensus.cc:729] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4bf6b0bc984d66856c3ad71c7ead83, State: Initialized, Role: FOLLOWER
I20260619 10:26:19.953105 16940 consensus_queue.cc:260] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:19.953333 16940 ts_tablet_manager.cc:1428] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:26:19.953344 16912 heartbeater.cc:503] Master 127.9.90.126:33979 was elected leader, sending a full tablet report...
I20260619 10:26:19.983282 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.980752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.0.0.1:41266)
0619 10:26:19.980807 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.980817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.980853 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.980997 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.980998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.980999 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:19.981043 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.981044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.981718 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.981718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.982487 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.982490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.983074 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.983075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.983127 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.983130 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.983184 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.983224 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 10:26:19.983279 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.980822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.9.90.124:35145)
0619 10:26:19.980869 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.980871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.980879 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:19.980902 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.980903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.980925 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.980990 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:19.981124 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.981655 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.982569 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.983003 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.983074 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.983138 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.983190 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.983219 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:26:19.983770 16468 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260619 10:26:19.984615 16497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b360c5f4e3437d8dee672b25e2dba4. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.984719 16497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: This master's current role is: LEARNER
I20260619 10:26:19.990044 16944 raft_consensus.cc:482] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:19.990147 16944 raft_consensus.cc:504] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:19.990592 16944 leader_election.cc:288] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c8715c390f407da0848f3d08206803 (127.9.90.65:37485), ad4bf6b0bc984d66856c3ad71c7ead83 (127.9.90.67:45345)
I20260619 10:26:19.994158 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.990804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:53975 (local address 127.9.90.67:45345)
0619 10:26:19.990845 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.990847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.990858 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:26:19.990886 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.990887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.990898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.990966 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:19.991169 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.991891 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.993246 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.993823 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.993896 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.993906 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.994057 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.994092 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:19.994158 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.990804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:56845 (local address 127.9.90.65:37485)
0619 10:26:19.990875 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:19.990876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:19.990883 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:19.990905 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:19.990905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:19.990915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:19.990966 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:19.991169 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.991898 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.993246 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.993840 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.993896 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.993906 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:19.994057 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 10:26:19.994091 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 10:26:19.994168 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.990738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:45345 (local address 127.9.90.66:53975)
0619 10:26:19.990791 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.990801 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.990838 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.990975 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.990976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.990978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:19.991065 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.991067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.991963 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.991964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.993158 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.993161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.993895 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.993896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.993989 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.993993 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.994047 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.994082 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":86}
I20260619 10:26:19.994168 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:19.990763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:37485 (local address 127.9.90.66:56845)
0619 10:26:19.990800 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:19.990807 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:19.990838 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:19.990976 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:19.990976 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:19.990978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:19.991075 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.991076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.991964 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.991964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.993158 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:19.993161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:19.993896 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:19.993896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:19.993989 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:19.993993 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:19.994047 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:26:19.994082 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":32}
I20260619 10:26:19.994556 16865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abf63ed99ee4a95874e58d1924996a4" candidate_uuid: "ad5e370ed46448198c12796120515e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" is_pre_election: true
I20260619 10:26:19.994742 16865 raft_consensus.cc:2454] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad5e370ed46448198c12796120515e87 in term 0.
I20260619 10:26:19.994728 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abf63ed99ee4a95874e58d1924996a4" candidate_uuid: "ad5e370ed46448198c12796120515e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c8715c390f407da0848f3d08206803" is_pre_election: true
I20260619 10:26:19.994879 16593 raft_consensus.cc:2454] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad5e370ed46448198c12796120515e87 in term 0.
I20260619 10:26:19.994989 16663 leader_election.cc:302] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad4bf6b0bc984d66856c3ad71c7ead83, ad5e370ed46448198c12796120515e87; no voters:
I20260619 10:26:19.995095 16944 raft_consensus.cc:2781] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:19.995146 16944 raft_consensus.cc:482] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:19.995172 16944 raft_consensus.cc:3037] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:19.995803 16944 raft_consensus.cc:504] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:19.995947 16944 leader_election.cc:288] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [CANDIDATE]: Term 1 election: Requested vote from peers 83c8715c390f407da0848f3d08206803 (127.9.90.65:37485), ad4bf6b0bc984d66856c3ad71c7ead83 (127.9.90.67:45345)
I20260619 10:26:19.996078 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abf63ed99ee4a95874e58d1924996a4" candidate_uuid: "ad5e370ed46448198c12796120515e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c8715c390f407da0848f3d08206803"
I20260619 10:26:19.996099 16865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abf63ed99ee4a95874e58d1924996a4" candidate_uuid: "ad5e370ed46448198c12796120515e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83"
I20260619 10:26:19.996151 16593 raft_consensus.cc:3037] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:19.996166 16865 raft_consensus.cc:3037] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:19.996795 16865 raft_consensus.cc:2454] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad5e370ed46448198c12796120515e87 in term 1.
I20260619 10:26:19.996802 16593 raft_consensus.cc:2454] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad5e370ed46448198c12796120515e87 in term 1.
I20260619 10:26:19.996932 16662 leader_election.cc:302] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c8715c390f407da0848f3d08206803, ad5e370ed46448198c12796120515e87; no voters:
I20260619 10:26:19.997040 16944 raft_consensus.cc:2781] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:19.997174 16944 raft_consensus.cc:686] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 1 LEADER]: Becoming Leader. State: Replica: ad5e370ed46448198c12796120515e87, State: Running, Role: LEADER
I20260619 10:26:19.997253 16944 consensus_queue.cc:237] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:19.997989 15673 catalog_manager.cc:5495] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad5e370ed46448198c12796120515e87 (127.9.90.66). New cstate: current_term: 1 leader_uuid: "ad5e370ed46448198c12796120515e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } health_report { overall_health: UNKNOWN } } }
I20260619 10:26:19.998596 16423 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: NON_VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: true } } }
I20260619 10:26:19.998683 16423 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:19.999387 16952 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: attempting to promote NON_VOTER 69c3bb4a9f0b445394a1b9a5d39bfcea to VOTER
I20260619 10:26:19.999511 16952 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } }
I20260619 10:26:19.999902 16468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 LEARNER]: Refusing update from remote peer e9b360c5f4e3437d8dee672b25e2dba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 10:26:20.000087 16281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Refusing update from remote peer e9b360c5f4e3437d8dee672b25e2dba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 10:26:20.000102 16954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260619 10:26:20.000270 16954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260619 10:26:20.000784 16951 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 69c3bb4a9f0b445394a1b9a5d39bfcea (127.9.90.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:20.000926 16468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 69c3bb4a9f0b445394a1b9a5d39bfcea (127.9.90.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:20.000979 16281 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 69c3bb4a9f0b445394a1b9a5d39bfcea (127.9.90.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:20.001775 16951 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:20.001816 16497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:20.001870 16951 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:20.001888 16497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:26:20.001969 16424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:20.002022 16424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:20.002094 16951 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:20.002140 16951 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:20.002097 16955 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:20.002184 16955 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:26:20.014511 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:20.012184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.9.90.66:45495)
0619 10:26:20.012236 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:20.012238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:20.012247 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:26:20.012326 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:20.012327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:20.012348 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:20.012410 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:20.012520 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:20.013003 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:20.013734 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:20.013988 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:20.014038 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:20.014240 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:20.014347 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:20.014418 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:20.014431 (+ 13us) server_negotiation.cc:299] Negotiation successful
0619 10:26:20.014451 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:26:20.016438 16593 raft_consensus.cc:1261] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [term 1 FOLLOWER]: Refusing update from remote peer ad5e370ed46448198c12796120515e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:20.016438 16865 raft_consensus.cc:1261] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 1 FOLLOWER]: Refusing update from remote peer ad5e370ed46448198c12796120515e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:20.016773 16949 consensus_queue.cc:1035] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:20.016898 16944 consensus_queue.cc:1035] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:20.020449 16958 mvcc.cc:205] Tried to move back new op lower bound from 7298518138944585728 to 7298518138869280768. Current Snapshot: MvccSnapshot[applied={T|T < 7298518138944585728}]
I20260619 10:26:20.022181 15672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:20.022291 15672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:20.023475 15672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9b360c5f4e3437d8dee672b25e2dba4: Sending DeleteTablet for 3 replicas of tablet 4abf63ed99ee4a95874e58d1924996a4
I20260619 10:26:20.023815 16845 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abf63ed99ee4a95874e58d1924996a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:20 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:39862
I20260619 10:26:20.023890 16573 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abf63ed99ee4a95874e58d1924996a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:20 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:52330
I20260619 10:26:20.024139 16968 tablet_replica.cc:330] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: stopping tablet replica
I20260619 10:26:20.024135 16709 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abf63ed99ee4a95874e58d1924996a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:20 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:45858
I20260619 10:26:20.024276 16968 raft_consensus.cc:2229] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:20.024407 16968 raft_consensus.cc:2258] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:20.024446 16969 tablet_replica.cc:330] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: stopping tablet replica
I20260619 10:26:20.024549 16969 raft_consensus.cc:2229] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:20.024650 16969 raft_consensus.cc:2258] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:20.024956 16968 ts_tablet_manager.cc:1905] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:20.025079 16969 ts_tablet_manager.cc:1905] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:20.025955 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 15641
I20260619 10:26:20.026894 16968 ts_tablet_manager.cc:1918] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:20.026993 16968 log.cc:1196] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/4abf63ed99ee4a95874e58d1924996a4
I20260619 10:26:20.027261 16968 ts_tablet_manager.cc:1939] T 4abf63ed99ee4a95874e58d1924996a4 P 83c8715c390f407da0848f3d08206803: Deleting consensus metadata
I20260619 10:26:20.027269 16969 ts_tablet_manager.cc:1918] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:20.027348 16969 log.cc:1196] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/4abf63ed99ee4a95874e58d1924996a4
I20260619 10:26:20.027591 16969 ts_tablet_manager.cc:1939] T 4abf63ed99ee4a95874e58d1924996a4 P ad5e370ed46448198c12796120515e87: Deleting consensus metadata
I20260619 10:26:20.029930 16967 tablet_replica.cc:330] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: stopping tablet replica
I20260619 10:26:20.030058 16967 raft_consensus.cc:2229] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:20.030182 16967 raft_consensus.cc:2258] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:20.030581 16967 ts_tablet_manager.cc:1905] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 10:26:20.031262 16662 connection.cc:535] server connection from 127.0.0.1:45858 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 10:26:20.031454 16529 connection.cc:535] server connection from 127.0.0.1:52330 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:26:20.032110 16967 ts_tablet_manager.cc:1918] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:20.032181 16967 log.cc:1196] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/4abf63ed99ee4a95874e58d1924996a4
I20260619 10:26:20.032421 16967 ts_tablet_manager.cc:1939] T 4abf63ed99ee4a95874e58d1924996a4 P ad4bf6b0bc984d66856c3ad71c7ead83: Deleting consensus metadata
W20260619 10:26:20.032944 16799 connection.cc:441] server connection from 127.0.0.1:39862 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39862 (request call id 1) could send its response
I20260619 10:26:20.039512 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:20.033171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.9.90.124:35145)
0619 10:26:20.033215 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:20.033216 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:20.033224 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:20.033309 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:20.033311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:20.033322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:20.033397 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:20.034096 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:20.034792 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:20.035465 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:20.035670 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:20.035724 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:20.036991 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:20.036993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:20.036993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:20.037000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:20.038528 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:20.038829 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:20.038830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:20.038831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:20.038886 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:20.039046 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:20.039047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:20.039048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:20.039156 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:20.039245 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:20.039339 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 10:26:20.039415 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:26:20.040642 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:20.032893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.9.90.125:46451)
0619 10:26:20.032935 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:20.032937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:20.033096 (+ 159us) server_negotiation.cc:407] Connection header received
0619 10:26:20.033327 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:20.033328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:20.033341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:20.033391 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:20.033943 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:20.034698 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:20.035771 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:20.036070 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:20.036122 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:20.037486 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:20.037487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:20.037488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:20.037496 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:20.039224 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:20.039900 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:20.039901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:20.039901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:20.039961 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:20.040166 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:20.040167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:20.040168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:20.040304 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:20.040391 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:20.040511 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 10:26:20.040580 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:21.020666 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.020397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.66:33653)
0619 10:26:21.020519 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.020609 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
W20260619 10:26:21.020987 16776 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:33979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
I20260619 10:26:21.021631 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.021450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.67:40925)
0619 10:26:21.021539 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.021596 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":54,"threads_started":1}
W20260619 10:26:21.021870 16912 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:33979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
I20260619 10:26:21.022078 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.021859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.65:56411)
0619 10:26:21.021969 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.022036 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":51,"threads_started":1}
W20260619 10:26:21.022264 16640 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:33979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
I20260619 10:26:21.523631 16979 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e9b360c5f4e3437d8dee672b25e2dba4)
I20260619 10:26:21.523761 16979 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } }
I20260619 10:26:21.524041 16979 leader_election.cc:288] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9b360c5f4e3437d8dee672b25e2dba4 (127.9.90.126:33979), 69c3bb4a9f0b445394a1b9a5d39bfcea (127.9.90.124:35145)
I20260619 10:26:21.524371 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.524148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:48060)
0619 10:26:21.524276 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.524327 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":42,"threads_started":1}
W20260619 10:26:21.524544 16257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
I20260619 10:26:21.524645 16983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e9b360c5f4e3437d8dee672b25e2dba4)
I20260619 10:26:21.524791 16983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } }
I20260619 10:26:21.524957 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.524869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:48070)
0619 10:26:21.524908 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.524931 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 10:26:21.525027 16257 leader_election.cc:334] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9b360c5f4e3437d8dee672b25e2dba4 (127.9.90.126:33979): Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
I20260619 10:26:21.525179 16983 leader_election.cc:288] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9b360c5f4e3437d8dee672b25e2dba4 (127.9.90.126:33979), ec6ee35d05ec4b0ba455a4eadeefd96a (127.9.90.125:46451)
I20260619 10:26:21.525274 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.525079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:48078)
0619 10:26:21.525179 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.525236 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":47,"threads_started":1}
W20260619 10:26:21.525368 16444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
I20260619 10:26:21.525758 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.525679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:48088)
0619 10:26:21.525711 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.525733 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260619 10:26:21.525839 16444 leader_election.cc:334] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9b360c5f4e3437d8dee672b25e2dba4 (127.9.90.126:33979): Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
I20260619 10:26:21.527374 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.524320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.9.90.124:35145)
0619 10:26:21.524428 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:21.524430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:21.524438 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:21.524465 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:21.524466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:21.524476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:21.524537 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:21.524767 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.525526 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.526692 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.527094 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.527148 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.527156 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:21.527305 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 10:26:21.527332 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
I20260619 10:26:21.527405 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.524196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:35145 (local address 127.0.0.1:41288)
0619 10:26:21.524286 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.524296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:21.524348 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:21.524602 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:21.524603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:21.524605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:21.524669 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.524672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.525617 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.525618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.526559 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.526562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.527167 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.527168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.527229 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.527233 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:21.527298 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 10:26:21.527338 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":41,"threads_started":1}
I20260619 10:26:21.527657 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" is_pre_election: true
I20260619 10:26:21.527756 16468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec6ee35d05ec4b0ba455a4eadeefd96a in term 1.
I20260619 10:26:21.527908 16258 leader_election.cc:302] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69c3bb4a9f0b445394a1b9a5d39bfcea, ec6ee35d05ec4b0ba455a4eadeefd96a; no voters: e9b360c5f4e3437d8dee672b25e2dba4
I20260619 10:26:21.528019 16979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 10:26:21.528062 16979 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e9b360c5f4e3437d8dee672b25e2dba4)
I20260619 10:26:21.528088 16979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:26:21.528383 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.525352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.9.90.125:46451)
0619 10:26:21.525467 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:21.525468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:21.525588 (+ 120us) server_negotiation.cc:407] Connection header received
0619 10:26:21.525710 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:21.525711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:21.525721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:21.525921 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:21.525958 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.526412 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.527584 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.528094 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.528157 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.528168 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:21.528290 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 10:26:21.528326 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":76,"threads_started":1}
I20260619 10:26:21.528394 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.525277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46451 (local address 127.0.0.1:34048)
0619 10:26:21.525509 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.525546 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:21.525609 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:21.525822 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:21.525823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:21.525826 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:21.525866 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.525867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.526484 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.526485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.527521 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.527524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.528167 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.528168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.528231 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.528235 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:21.528281 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:26:21.528319 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":41,"threads_started":1}
I20260619 10:26:21.528621 16281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" is_pre_election: true
I20260619 10:26:21.528877 16979 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } }
I20260619 10:26:21.528997 16281 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69c3bb4a9f0b445394a1b9a5d39bfcea in current term 2: Already voted for candidate ec6ee35d05ec4b0ba455a4eadeefd96a in this term.
I20260619 10:26:21.529001 16979 leader_election.cc:288] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [CANDIDATE]: Term 2 election: Requested vote from peers e9b360c5f4e3437d8dee672b25e2dba4 (127.9.90.126:33979), 69c3bb4a9f0b445394a1b9a5d39bfcea (127.9.90.124:35145)
I20260619 10:26:21.529189 16442 leader_election.cc:302] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69c3bb4a9f0b445394a1b9a5d39bfcea; no voters: e9b360c5f4e3437d8dee672b25e2dba4, ec6ee35d05ec4b0ba455a4eadeefd96a
I20260619 10:26:21.529223 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.529077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:48100)
0619 10:26:21.529183 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.529208 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 10:26:21.529171 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea"
I20260619 10:26:21.529310 16983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:26:21.530005 16983 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:26:21.530087 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.529993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:48102)
0619 10:26:21.530036 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.530066 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:26:21.530146 16257 leader_election.cc:334] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e9b360c5f4e3437d8dee672b25e2dba4 (127.9.90.126:33979): Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
I20260619 10:26:21.530640 16468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6ee35d05ec4b0ba455a4eadeefd96a in term 2.
I20260619 10:26:21.530783 16258 leader_election.cc:302] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69c3bb4a9f0b445394a1b9a5d39bfcea, ec6ee35d05ec4b0ba455a4eadeefd96a; no voters: e9b360c5f4e3437d8dee672b25e2dba4
I20260619 10:26:21.530871 16979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:26:21.530998 16979 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [term 2 LEADER]: Becoming Leader. State: Replica: ec6ee35d05ec4b0ba455a4eadeefd96a, State: Running, Role: LEADER
I20260619 10:26:21.531071 16979 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } }
I20260619 10:26:21.531391 16989 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec6ee35d05ec4b0ba455a4eadeefd96a. Latest consensus state: current_term: 2 leader_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:21.531479 16989 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:21.531641 16991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:21.531978 16991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:21.532032 16991 catalog_manager.cc:1259] Loaded cluster ID: d1f382c22a204f1b8f634281311475a0
I20260619 10:26:21.532051 16991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:21.532155 16991 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:21.532205 16991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Loaded TSK: 0
I20260619 10:26:21.532312 16991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:26:21.570910 16271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:21.572467 16271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:21.573422 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.573313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:48110)
0619 10:26:21.573357 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.573391 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 10:26:21.573439 16468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [term 2 FOLLOWER]: Refusing update from remote peer ec6ee35d05ec4b0ba455a4eadeefd96a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 10:26:21.573765 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.573667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.0.0.1:48118)
0619 10:26:21.573707 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.573748 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 10:26:21.573716 16989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 10:26:21.573859 16257 consensus_peers.cc:487] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a -> Peer e9b360c5f4e3437d8dee672b25e2dba4 (127.9.90.126:33979): Couldn't send request to peer e9b360c5f4e3437d8dee672b25e2dba4. Status: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 10:26:21.574362 16983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec6ee35d05ec4b0ba455a4eadeefd96a. Latest consensus state: current_term: 2 leader_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:21.574448 16983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:26:21.574697 16992 mvcc.cc:205] Tried to move back new op lower bound from 7298518145322938368 to 7298518145151758336. Current Snapshot: MvccSnapshot[applied={T|T < 7298518145322938368}]
I20260619 10:26:21.574810 16989 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:21.574882 16989 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:21.575029 16983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e9b360c5f4e3437d8dee672b25e2dba4" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33979 } } peers { permanent_uuid: "ec6ee35d05ec4b0ba455a4eadeefd96a" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46451 } attrs { promote: false } } peers { permanent_uuid: "69c3bb4a9f0b445394a1b9a5d39bfcea" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 35145 } attrs { promote: false } } }
I20260619 10:26:21.575119 16983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c3bb4a9f0b445394a1b9a5d39bfcea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:26:21.575186 16993 mvcc.cc:205] Tried to move back new op lower bound from 7298518145322938368 to 7298518145151758336. Current Snapshot: MvccSnapshot[applied={T|T < 7298518145322938368 or (T in {7298518145322938368})}]
I20260619 10:26:21.579826 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.576537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45495 (local address 127.0.0.1:45882)
0619 10:26:21.576571 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.576580 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:21.576613 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:21.577038 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:21.577039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:21.577040 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:21.577064 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.577065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.578000 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.578001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.579029 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.579031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.579623 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.579624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.579693 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.579698 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:26:21.579727 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 10:26:21.579766 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 10:26:21.579960 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.576607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:37485 (local address 127.0.0.1:52336)
0619 10:26:21.576885 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.576893 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:21.576940 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:21.577130 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:21.577131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:21.577132 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:21.577186 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.577188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.577832 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.577833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.578964 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.578966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.579729 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.579730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.579797 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.579800 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:21.579859 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:26:21.579896 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":164,"thread_start_us":76,"threads_started":1}
I20260619 10:26:21.579974 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.576689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.9.90.65:37485)
0619 10:26:21.576810 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:21.576812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:21.576939 (+ 127us) server_negotiation.cc:407] Connection header received
0619 10:26:21.577022 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:21.577023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:21.577034 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:21.577087 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:21.577255 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.577743 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.579057 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.579643 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.579715 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.579726 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:21.579869 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 10:26:21.579905 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260619 10:26:21.580230 16573 tablet_service.cc:1467] Processing CreateTablet for tablet fa679f4f6fe545f3bb6db2325ecfcc84 (DEFAULT_TABLE table=test-table [id=d7934336723d4520927e8c31a632dfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:21.580330 16573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa679f4f6fe545f3bb6db2325ecfcc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:21.580818 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.576604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.9.90.66:45495)
0619 10:26:21.576917 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:21.576920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:21.576927 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:21.576949 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:21.576950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:21.576960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:21.577120 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:21.577153 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.577926 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.579102 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.579532 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.580670 (+ 1138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.580688 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:21.580708 (+ 20us) server_negotiation.cc:299] Negotiation successful
0619 10:26:21.580749 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":61,"threads_started":1}
I20260619 10:26:21.581051 16709 tablet_service.cc:1467] Processing CreateTablet for tablet fa679f4f6fe545f3bb6db2325ecfcc84 (DEFAULT_TABLE table=test-table [id=d7934336723d4520927e8c31a632dfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:21.581071 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.576607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:45345 (local address 127.0.0.1:39870)
0619 10:26:21.576701 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.576710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:21.576767 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:21.576987 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:21.576988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:21.576990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:21.577042 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.577044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.579095 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.579096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.580110 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.580112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.580852 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.580853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.580914 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.580918 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:21.580973 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 10:26:21.581015 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 10:26:21.581182 16709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa679f4f6fe545f3bb6db2325ecfcc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:21.581256 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.576711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.9.90.67:45345)
0619 10:26:21.576858 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:21.576860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:21.576867 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:21.576893 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:21.576894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:21.576904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:21.577100 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:21.577135 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.577935 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.580188 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.580769 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.580830 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.580842 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:21.581147 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 10:26:21.581191 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":56,"threads_started":1}
I20260619 10:26:21.581540 17001 tablet_bootstrap.cc:492] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803: Bootstrap starting.
I20260619 10:26:21.581954 17001 tablet_bootstrap.cc:654] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:21.582022 16845 tablet_service.cc:1467] Processing CreateTablet for tablet fa679f4f6fe545f3bb6db2325ecfcc84 (DEFAULT_TABLE table=test-table [id=d7934336723d4520927e8c31a632dfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:21.582135 16845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa679f4f6fe545f3bb6db2325ecfcc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:21.582572 17003 tablet_bootstrap.cc:492] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: Bootstrap starting.
I20260619 10:26:21.582935 17003 tablet_bootstrap.cc:654] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:21.583160 17004 tablet_bootstrap.cc:492] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83: Bootstrap starting.
I20260619 10:26:21.583334 17001 tablet_bootstrap.cc:492] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803: No bootstrap required, opened a new log
I20260619 10:26:21.583413 17001 ts_tablet_manager.cc:1397] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:21.583454 17003 tablet_bootstrap.cc:492] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: No bootstrap required, opened a new log
I20260619 10:26:21.583467 17004 tablet_bootstrap.cc:654] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:21.583503 17003 ts_tablet_manager.cc:1397] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:21.583606 17003 raft_consensus.cc:348] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:21.583606 17001 raft_consensus.cc:348] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:21.583670 17003 raft_consensus.cc:374] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:21.583690 17003 raft_consensus.cc:729] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad5e370ed46448198c12796120515e87, State: Initialized, Role: FOLLOWER
I20260619 10:26:21.583707 17001 raft_consensus.cc:374] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:21.583745 17003 consensus_queue.cc:260] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:21.583768 17001 raft_consensus.cc:729] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c8715c390f407da0848f3d08206803, State: Initialized, Role: FOLLOWER
I20260619 10:26:21.583856 17001 consensus_queue.cc:260] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:21.583881 17003 ts_tablet_manager.cc:1428] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:21.583946 17004 tablet_bootstrap.cc:492] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83: No bootstrap required, opened a new log
I20260619 10:26:21.583998 17004 ts_tablet_manager.cc:1397] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:21.584053 17001 ts_tablet_manager.cc:1428] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 10:26:21.584128 17004 raft_consensus.cc:348] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:21.584275 17004 raft_consensus.cc:374] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:21.584538 17004 raft_consensus.cc:729] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4bf6b0bc984d66856c3ad71c7ead83, State: Initialized, Role: FOLLOWER
I20260619 10:26:21.584669 17004 consensus_queue.cc:260] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:21.584872 16777 heartbeater.cc:503] Master 127.9.90.125:46451 was elected leader, sending a full tablet report...
I20260619 10:26:21.584935 17004 ts_tablet_manager.cc:1428] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:21.585359 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.585118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.67:49437)
0619 10:26:21.585281 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.585317 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260619 10:26:21.585680 16913 heartbeater.cc:503] Master 127.9.90.125:46451 was elected leader, sending a full tablet report...
I20260619 10:26:21.585747 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.585579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.67:53629)
0619 10:26:21.585654 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.585678 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 10:26:21.586038 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.584121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.66:49831)
0619 10:26:21.585300 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.585336 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":67,"threads_started":1}
I20260619 10:26:21.586272 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.584456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.65:47041)
0619 10:26:21.586202 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.586238 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1726,"thread_start_us":68,"threads_started":1}
I20260619 10:26:21.586341 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.586256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.66:39905)
0619 10:26:21.586288 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.586314 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 10:26:21.586465 16641 heartbeater.cc:503] Master 127.9.90.125:46451 was elected leader, sending a full tablet report...
I20260619 10:26:21.586617 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.586538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.65:48085)
0619 10:26:21.586569 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.586595 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260619 10:26:21.651886 16642 tablet.cc:2369] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:26:21.664985 16914 tablet.cc:2369] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:26:21.790305 16778 tablet.cc:2369] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:26:21.878800 17011 raft_consensus.cc:482] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:21.878901 17011 raft_consensus.cc:504] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:21.879254 17011 leader_election.cc:288] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad5e370ed46448198c12796120515e87 (127.9.90.66:45495), 83c8715c390f407da0848f3d08206803 (127.9.90.65:37485)
I20260619 10:26:21.883095 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.879535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:45719 (local address 127.9.90.65:37485)
0619 10:26:21.879585 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:21.879586 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:21.879850 (+ 264us) server_negotiation.cc:407] Connection header received
0619 10:26:21.879877 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:21.879878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:21.879890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:21.879939 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:21.880057 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.880735 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.882124 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.882703 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.882803 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.882814 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:21.882990 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 10:26:21.883026 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:26:21.883113 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.879503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:37159 (local address 127.9.90.66:45495)
0619 10:26:21.879551 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:21.879552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:21.879601 (+ 49us) server_negotiation.cc:407] Connection header received
0619 10:26:21.879630 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:21.879631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:21.879642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:21.879689 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:21.879853 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.880612 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.882124 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.882703 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.882803 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.882815 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:21.882979 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 10:26:21.883058 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:26:21.883102 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.879439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45495 (local address 127.9.90.67:37159)
0619 10:26:21.879486 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.879498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:21.879547 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:21.879710 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:21.879712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:21.879715 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:21.879784 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.879786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.880693 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.880694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.882016 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.882018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.882815 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.882816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.882901 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.882907 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 10:26:21.882967 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 10:26:21.883012 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":59}
I20260619 10:26:21.883108 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.879439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:37485 (local address 127.9.90.67:45719)
0619 10:26:21.879629 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.879637 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:21.879680 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:21.879948 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:21.879948 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:21.879950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:21.880000 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.880002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.880804 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.880805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.882016 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:21.882018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.882845 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.882846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:21.882902 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.882908 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 10:26:21.882978 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 10:26:21.883012 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":60,"thread_start_us":104,"threads_started":1}
I20260619 10:26:21.883381 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa679f4f6fe545f3bb6db2325ecfcc84" candidate_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c8715c390f407da0848f3d08206803" is_pre_election: true
I20260619 10:26:21.883479 16593 raft_consensus.cc:2454] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad4bf6b0bc984d66856c3ad71c7ead83 in term 0.
I20260619 10:26:21.883433 16729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa679f4f6fe545f3bb6db2325ecfcc84" candidate_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad5e370ed46448198c12796120515e87" is_pre_election: true
I20260619 10:26:21.883585 16729 raft_consensus.cc:2454] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad4bf6b0bc984d66856c3ad71c7ead83 in term 0.
I20260619 10:26:21.883667 16798 leader_election.cc:302] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c8715c390f407da0848f3d08206803, ad4bf6b0bc984d66856c3ad71c7ead83; no voters:
I20260619 10:26:21.883817 17011 raft_consensus.cc:2781] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:21.883859 17011 raft_consensus.cc:482] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:21.883873 17011 raft_consensus.cc:3037] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:21.884521 17011 raft_consensus.cc:504] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:21.884647 17011 leader_election.cc:288] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [CANDIDATE]: Term 1 election: Requested vote from peers ad5e370ed46448198c12796120515e87 (127.9.90.66:45495), 83c8715c390f407da0848f3d08206803 (127.9.90.65:37485)
I20260619 10:26:21.884805 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa679f4f6fe545f3bb6db2325ecfcc84" candidate_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c8715c390f407da0848f3d08206803"
I20260619 10:26:21.884804 16729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa679f4f6fe545f3bb6db2325ecfcc84" candidate_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad5e370ed46448198c12796120515e87"
I20260619 10:26:21.884869 16593 raft_consensus.cc:3037] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:21.884869 16729 raft_consensus.cc:3037] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:21.885406 16593 raft_consensus.cc:2454] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad4bf6b0bc984d66856c3ad71c7ead83 in term 1.
I20260619 10:26:21.885560 16798 leader_election.cc:302] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c8715c390f407da0848f3d08206803, ad4bf6b0bc984d66856c3ad71c7ead83; no voters:
I20260619 10:26:21.885566 16729 raft_consensus.cc:2454] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad4bf6b0bc984d66856c3ad71c7ead83 in term 1.
I20260619 10:26:21.885645 17011 raft_consensus.cc:2781] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:21.885813 17011 raft_consensus.cc:686] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 1 LEADER]: Becoming Leader. State: Replica: ad4bf6b0bc984d66856c3ad71c7ead83, State: Running, Role: LEADER
I20260619 10:26:21.885886 17011 consensus_queue.cc:237] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } }
I20260619 10:26:21.886183 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.886084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.67:41103)
0619 10:26:21.886117 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.886153 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 10:26:21.886665 16271 catalog_manager.cc:5495] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad4bf6b0bc984d66856c3ad71c7ead83 (127.9.90.67). New cstate: current_term: 1 leader_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad4bf6b0bc984d66856c3ad71c7ead83" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 45345 } health_report { overall_health: HEALTHY } } }
I20260619 10:26:21.931947 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.929684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.9.90.67:45345)
0619 10:26:21.929753 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:21.929754 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:21.929761 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:21.929809 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:21.929810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:21.929826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:21.929888 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:21.930019 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.930489 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.931236 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:21.931421 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:21.931473 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:21.931623 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:21.931723 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:21.931785 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:21.931825 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 10:26:21.931865 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 10:26:21.933051 16593 raft_consensus.cc:1261] T fa679f4f6fe545f3bb6db2325ecfcc84 P 83c8715c390f407da0848f3d08206803 [term 1 FOLLOWER]: Refusing update from remote peer ad4bf6b0bc984d66856c3ad71c7ead83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:21.933107 16729 raft_consensus.cc:1261] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [term 1 FOLLOWER]: Refusing update from remote peer ad4bf6b0bc984d66856c3ad71c7ead83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:21.933337 17014 consensus_queue.cc:1035] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c8715c390f407da0848f3d08206803" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:21.933461 17011 consensus_queue.cc:1035] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e370ed46448198c12796120515e87" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:21.934250 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.934163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.66:60945)
0619 10:26:21.934203 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.934223 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 10:26:21.934255 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.934092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.65:51873)
0619 10:26:21.934179 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.934211 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 10:26:21.935585 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.935318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.67:60575)
0619 10:26:21.935495 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.935533 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 10:26:21.935796 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.935715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.65:58377)
0619 10:26:21.935747 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.935771 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 10:26:21.935869 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.935779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33979 (local address 127.9.90.66:40665)
0619 10:26:21.935815 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.935840 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 10:26:21.936127 17017 mvcc.cc:205] Tried to move back new op lower bound from 7298518146795597824 to 7298518146605060096. Current Snapshot: MvccSnapshot[applied={T|T < 7298518146795597824 or (T in {7298518146795597824})}]
I20260619 10:26:21.936836 17020 mvcc.cc:205] Tried to move back new op lower bound from 7298518146795597824 to 7298518146605060096. Current Snapshot: MvccSnapshot[applied={T|T < 7298518146795597824 or (T in {7298518146795597824})}]
I20260619 10:26:21.937841 16271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:21.937920 16271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:21.939044 16271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec6ee35d05ec4b0ba455a4eadeefd96a: Sending DeleteTablet for 3 replicas of tablet fa679f4f6fe545f3bb6db2325ecfcc84
I20260619 10:26:21.939349 16573 tablet_service.cc:1514] Processing DeleteTablet for tablet fa679f4f6fe545f3bb6db2325ecfcc84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:21 UTC) from {username='oryx', principal='oryx/127.9.90.125@KRBTEST.COM'} at 127.0.0.1:52336
I20260619 10:26:21.939610 16845 tablet_service.cc:1514] Processing DeleteTablet for tablet fa679f4f6fe545f3bb6db2325ecfcc84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:21 UTC) from {username='oryx', principal='oryx/127.9.90.125@KRBTEST.COM'} at 127.0.0.1:39870
I20260619 10:26:21.939663 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 16510
I20260619 10:26:21.939677 16709 tablet_service.cc:1514] Processing DeleteTablet for tablet fa679f4f6fe545f3bb6db2325ecfcc84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:21 UTC) from {username='oryx', principal='oryx/127.9.90.125@KRBTEST.COM'} at 127.0.0.1:45882
I20260619 10:26:21.940798 17029 tablet_replica.cc:330] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: stopping tablet replica
I20260619 10:26:21.940850 17029 raft_consensus.cc:2229] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:21.940888 17029 raft_consensus.cc:2258] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:21.941156 17029 ts_tablet_manager.cc:1905] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:21.942798 17029 ts_tablet_manager.cc:1918] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:21.942854 17029 log.cc:1196] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/fa679f4f6fe545f3bb6db2325ecfcc84
I20260619 10:26:21.943074 17029 ts_tablet_manager.cc:1939] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad5e370ed46448198c12796120515e87: Deleting consensus metadata
I20260619 10:26:21.943377 16258 catalog_manager.cc:4841] TS ad5e370ed46448198c12796120515e87 (127.9.90.66:45495): tablet fa679f4f6fe545f3bb6db2325ecfcc84 (table test-table [id=d7934336723d4520927e8c31a632dfa3]) successfully deleted
I20260619 10:26:21.945704 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.945579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:37485 (local address 127.0.0.1:52346)
0619 10:26:21.945639 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.945670 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:37485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 10:26:21.945710 17028 tablet_replica.cc:330] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83: stopping tablet replica
I20260619 10:26:21.945775 17028 raft_consensus.cc:2229] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:21.945878 17028 raft_consensus.cc:2258] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 10:26:21.945878 16255 catalog_manager.cc:4584] TS 83c8715c390f407da0848f3d08206803 (127.9.90.65:37485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa679f4f6fe545f3bb6db2325ecfcc84: Network error: Client connection negotiation failed: client connection to 127.9.90.65:37485: connect: Connection refused (error 111)
I20260619 10:26:21.946015 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 16646
I20260619 10:26:21.950763 17028 ts_tablet_manager.cc:1905] T fa679f4f6fe545f3bb6db2325ecfcc84 P ad4bf6b0bc984d66856c3ad71c7ead83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:21.951390 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 16782
I20260619 10:26:21.956338 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 16137
I20260619 10:26:21.956468 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:21.956334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:45345 (local address 127.0.0.1:39878)
0619 10:26:21.956391 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:21.956431 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.67:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 10:26:21.960232 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 16325
2026-06-19T10:26:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4634 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:22 dist-test-slave-hw7q krb5kdc[17032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:22 dist-test-slave-hw7q krb5kdc[17032](info): set up 2 sockets
Jun 19 10:26:22 dist-test-slave-hw7q krb5kdc[17032](info): commencing operation
krb5kdc: starting...
W20260619 10:26:22.921876 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:22 dist-test-slave-hw7q krb5kdc[17032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.126.keytab.
Entry for principal oryx/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.126.keytab.
I20260619 10:26:23.030364 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:35887
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.126.keytab
--principal=oryx/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:35887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:23.107323 17045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:23.107519 17045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:23.107540 17045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:23.108623 17045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:23.108654 17045 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 10:26:23.108665 17045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:23.108677 17045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:23.108693 17045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:23.108704 17045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:23.110141 17045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:35887
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.126.keytab
--principal=oryx/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:23.110327 17045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:23.110487 17045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:23.114831 17053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:23.114802 17051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:23.114848 17045 server_base.cc:1034] running on GCE node
W20260619 10:26:23.114815 17050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:23.115299 17045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:23.115552 17045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:23.116685 17045 hybrid_clock.cc:648] HybridClock initialized: now 1781864783116672 us; error 29 us; skew 500 ppm
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:23.119159 17045 init.cc:377] Logged in from keytab as oryx/127.9.90.126@KRBTEST.COM (short username oryx)
I20260619 10:26:23.119565 17045 webserver.cc:460] Webserver started at http://127.9.90.126:41059/ using document root <none> and password file <none>
I20260619 10:26:23.119776 17045 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:23.119822 17045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:23.119941 17045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:23.120970 17045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "0f4cda0a95a14915b68059ca4f5b5c88"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "014f015551d5579196ee28290665462f"
server_key_iv: "4d471ce38a07b523c908ad50801b4b62"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.121285 17045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "0f4cda0a95a14915b68059ca4f5b5c88"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "014f015551d5579196ee28290665462f"
server_key_iv: "4d471ce38a07b523c908ad50801b4b62"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.122525 17045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:23.123129 17060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.123277 17045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:23.123358 17045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "0f4cda0a95a14915b68059ca4f5b5c88"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "014f015551d5579196ee28290665462f"
server_key_iv: "4d471ce38a07b523c908ad50801b4b62"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.123413 17045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:23.140017 17045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:23.141899 17045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:23.142037 17045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:23.146250 17045 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:35887
I20260619 10:26:23.146320 17122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:35887 every 8 connection(s)
I20260619 10:26:23.146606 17045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:23.147230 17123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.149374 17123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88: Bootstrap starting.
I20260619 10:26:23.149946 17123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.150208 17123 log.cc:824] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:23.150820 17123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88: No bootstrap required, opened a new log
I20260619 10:26:23.152733 17123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cda0a95a14915b68059ca4f5b5c88" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35887 } }
I20260619 10:26:23.152864 17123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.152895 17123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4cda0a95a14915b68059ca4f5b5c88, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.153034 17123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cda0a95a14915b68059ca4f5b5c88" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35887 } }
I20260619 10:26:23.153105 17123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:23.153139 17123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:23.153184 17123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:23.153661 17123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cda0a95a14915b68059ca4f5b5c88" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35887 } }
I20260619 10:26:23.153745 17123 leader_election.cc:302] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f4cda0a95a14915b68059ca4f5b5c88; no voters:
I20260619 10:26:23.153910 17123 leader_election.cc:288] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:23.153949 17126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:23.154075 17126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [term 1 LEADER]: Becoming Leader. State: Replica: 0f4cda0a95a14915b68059ca4f5b5c88, State: Running, Role: LEADER
I20260619 10:26:23.154158 17123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:23.154161 17126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cda0a95a14915b68059ca4f5b5c88" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35887 } }
I20260619 10:26:23.154574 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 17045
I20260619 10:26:23.154553 17127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f4cda0a95a14915b68059ca4f5b5c88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cda0a95a14915b68059ca4f5b5c88" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35887 } } }
I20260619 10:26:23.154624 17127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:23.154661 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:23.154780 17128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4cda0a95a14915b68059ca4f5b5c88. Latest consensus state: current_term: 1 leader_uuid: "0f4cda0a95a14915b68059ca4f5b5c88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cda0a95a14915b68059ca4f5b5c88" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35887 } } }
I20260619 10:26:23.154835 17128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:23.154867 9577 external_mini_cluster.cc:1428] Setting key 2b652b7f7bff7dbbbcc402032c4f6c05
I20260619 10:26:23.154935 17134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:23.155452 17134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:23.156941 17134 catalog_manager.cc:1347] Generated new cluster ID: 59335aeeb45f478099aa0622a75d566d
I20260619 10:26:23.156992 17134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:23.164193 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.155450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.9.90.126:35887)
0619 10:26:23.155637 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.155647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.155786 (+ 139us) server_negotiation.cc:407] Connection header received
0619 10:26:23.156052 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.156060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.156120 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.156254 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.157010 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.157970 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.158699 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.159118 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.159206 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.161294 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:23.161298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:23.161299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:23.161311 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:23.162839 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.163148 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.163158 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.163158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.163237 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.163404 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.163405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.163406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.163625 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:23.163766 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.163857 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.163990 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":57,"threads_started":1}
I20260619 10:26:23.175452 17134 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:23.176002 17134 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:23.178659 17134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88: Generated new TSK 0
I20260619 10:26:23.178859 17134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab.
Entry for principal oryx/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab.
I20260619 10:26:23.202242 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:35887
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab
--principal=oryx/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:23.279464 17149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:23.279672 17149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:23.279695 17149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:23.280804 17149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:23.280845 17149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:23.280864 17149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:26:23.280879 17149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:23.282370 17149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab
--principal=oryx/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.90.126:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:23.282582 17149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:23.282775 17149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:23.287635 17155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:23.287624 17154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:23.287773 17157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:23.287815 17149 server_base.cc:1034] running on GCE node
I20260619 10:26:23.287988 17149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:23.288203 17149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:23.289351 17149 hybrid_clock.cc:648] HybridClock initialized: now 1781864783289335 us; error 28 us; skew 500 ppm
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:23.291790 17149 init.cc:377] Logged in from keytab as oryx/127.9.90.65@KRBTEST.COM (short username oryx)
I20260619 10:26:23.292172 17149 webserver.cc:460] Webserver started at http://127.9.90.65:33821/ using document root <none> and password file <none>
I20260619 10:26:23.292367 17149 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:23.292413 17149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:23.292533 17149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:23.293602 17149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "f19e52fbd5ee4117aeb7e18f41f2018c"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "e6cb5324e70403c0485b30b6abf49018"
server_key_iv: "a2202b1c38037f2b0abe6a5ca8bbcfad"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.293946 17149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "f19e52fbd5ee4117aeb7e18f41f2018c"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "e6cb5324e70403c0485b30b6abf49018"
server_key_iv: "a2202b1c38037f2b0abe6a5ca8bbcfad"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.295152 17149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:23.295900 17164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.296053 17149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:23.296131 17149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "f19e52fbd5ee4117aeb7e18f41f2018c"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "e6cb5324e70403c0485b30b6abf49018"
server_key_iv: "a2202b1c38037f2b0abe6a5ca8bbcfad"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.296190 17149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:23.318387 17149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:23.320232 17149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:23.320353 17149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:23.320839 17149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:23.320879 17149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.320900 17149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:23.320912 17149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.327087 17149 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:36801
I20260619 10:26:23.327155 17279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:36801 every 8 connection(s)
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:23.327462 17149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:23.329960 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.321261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:47343 (local address 127.9.90.126:35887)
0619 10:26:23.321331 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.321333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.321483 (+ 150us) server_negotiation.cc:407] Connection header received
0619 10:26:23.322030 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.322032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.322044 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.322119 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.322898 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.323640 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.325081 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.325409 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.325487 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.327741 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:23.327743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:23.327743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:23.327751 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:23.328863 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.329195 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.329196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.329196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.329238 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.329403 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.329403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.329403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.329492 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:23.329535 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.329831 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.329898 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:26:23.330150 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.321222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35887 (local address 127.9.90.65:47343)
0619 10:26:23.321362 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.321401 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.321938 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.322209 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.322212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.322532 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:23.322796 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.322803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.323752 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.323753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.324987 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.324990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.325491 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.325492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.325590 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.326056 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:23.326066 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:23.327631 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:23.328927 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.328933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.328934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.329139 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.329279 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.329280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.329280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.329363 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.329567 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:23.329573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:23.329688 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.329780 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.329950 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260619 10:26:23.330596 17280 heartbeater.cc:347] Connected to a master server at 127.9.90.126:35887
I20260619 10:26:23.330667 17280 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:23.330816 17280 heartbeater.cc:511] Master 127.9.90.126:35887 requested a full tablet report, sending...
I20260619 10:26:23.331278 17076 ts_manager.cc:194] Registered new tserver with Master: f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65:36801)
I20260619 10:26:23.331954 17076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:47343
I20260619 10:26:23.335938 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.331540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:60471 (local address 127.9.90.126:35887)
0619 10:26:23.331574 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.331575 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.331692 (+ 117us) server_negotiation.cc:407] Connection header received
0619 10:26:23.331740 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.331741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.331750 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.331792 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.332071 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.332591 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.333219 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.333386 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.333430 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.334129 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:23.334130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:23.334130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:23.334135 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:23.335244 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.335480 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.335480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.335481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.335520 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.335625 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.335625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.335625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.335703 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:23.335744 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.335831 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.335880 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:26:23.335945 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.331495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35887 (local address 127.9.90.65:60471)
0619 10:26:23.331628 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.331637 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.331689 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.331795 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.331795 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.331949 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:23.331995 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.331996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.332677 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.332678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.333153 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.333154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.333434 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.333435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.333476 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.333804 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:23.333808 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:23.334058 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:23.335302 (+ 1244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.335303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.335303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.335432 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.335554 (+ 122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.335554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.335554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.335594 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.335742 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:23.335743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:23.335774 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.335816 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.335883 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260619 10:26:23.336418 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 17149
I20260619 10:26:23.336508 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:23.336716 9577 external_mini_cluster.cc:1428] Setting key cce1790ecd2e29ea62711a9c81deba32
WARNING: no policy specified for oryx/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab.
Entry for principal oryx/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab.
I20260619 10:26:23.357481 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:35887
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab
--principal=oryx/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:23.434855 17289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:23.435051 17289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:23.435073 17289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:23.436196 17289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:23.436236 17289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:23.436256 17289 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:26:23.436285 17289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:23.437731 17289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab
--principal=oryx/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.90.126:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:23.437956 17289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:23.438158 17289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:23.442764 17297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:23.442806 17294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:23.442791 17295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:23.442843 17289 server_base.cc:1034] running on GCE node
I20260619 10:26:23.443099 17289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:23.443321 17289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:23.444499 17289 hybrid_clock.cc:648] HybridClock initialized: now 1781864783444482 us; error 26 us; skew 500 ppm
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:23.446832 17289 init.cc:377] Logged in from keytab as oryx/127.9.90.66@KRBTEST.COM (short username oryx)
I20260619 10:26:23.447224 17289 webserver.cc:460] Webserver started at http://127.9.90.66:41913/ using document root <none> and password file <none>
I20260619 10:26:23.447386 17289 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:23.447422 17289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:23.447507 17289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:23.448431 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "6e504c6ffbe34d30a5bd82392b652b0e"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "fedc71c2632075ab706d242ca7e0db44"
server_key_iv: "a40a0d340ddb9c7580d3abf9a3d0c482"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.448714 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "6e504c6ffbe34d30a5bd82392b652b0e"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "fedc71c2632075ab706d242ca7e0db44"
server_key_iv: "a40a0d340ddb9c7580d3abf9a3d0c482"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.449873 17289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:23.450582 17304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.450793 17289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:23.450877 17289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "6e504c6ffbe34d30a5bd82392b652b0e"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "fedc71c2632075ab706d242ca7e0db44"
server_key_iv: "a40a0d340ddb9c7580d3abf9a3d0c482"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.450941 17289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:23.480520 17289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:23.482473 17289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:23.482606 17289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:23.483083 17289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:23.483125 17289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.483155 17289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:23.483171 17289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:23.489753 17289 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:39659
I20260619 10:26:23.490099 17289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:23.490140 17419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:39659 every 8 connection(s)
I20260619 10:26:23.491314 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 17289
I20260619 10:26:23.491401 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:23.491580 9577 external_mini_cluster.cc:1428] Setting key d4f65be8490a5f815a470e068dcaf16e
I20260619 10:26:23.492118 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.483527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:55595 (local address 127.9.90.126:35887)
0619 10:26:23.483566 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.483568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.483810 (+ 242us) server_negotiation.cc:407] Connection header received
0619 10:26:23.484398 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.484399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.484414 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.484471 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.485330 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.486098 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.487076 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.487292 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.487349 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.489350 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:23.489351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:23.489352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:23.489360 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:23.490814 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.491172 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.491173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.491173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.491231 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.491474 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.491475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.491475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.491583 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:23.491643 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.491957 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.492037 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:26:23.492233 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.483533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35887 (local address 127.9.90.66:55595)
0619 10:26:23.483685 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.483720 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.484298 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.484583 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.484589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.484946 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:23.485232 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.485239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.486186 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.486187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.486993 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.486995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.487382 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.487383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.487512 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.487983 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:23.487990 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:23.489265 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:23.490885 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.490891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.490893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.491101 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.491284 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.491285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.491285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.491402 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.491648 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:23.491654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:23.491805 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.491913 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.492033 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260619 10:26:23.493583 17420 heartbeater.cc:347] Connected to a master server at 127.9.90.126:35887
I20260619 10:26:23.493770 17420 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:23.493974 17420 heartbeater.cc:511] Master 127.9.90.126:35887 requested a full tablet report, sending...
I20260619 10:26:23.494328 17077 ts_manager.cc:194] Registered new tserver with Master: 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659)
I20260619 10:26:23.494819 17077 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:55595
I20260619 10:26:23.498893 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.493588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:49167 (local address 127.9.90.126:35887)
0619 10:26:23.493652 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.493654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.493809 (+ 155us) server_negotiation.cc:407] Connection header received
0619 10:26:23.493861 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.493862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.493872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.493922 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.494256 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.494931 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.495658 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.495860 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.495913 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.496686 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:23.496687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:23.496688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:23.496694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:23.497904 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.498175 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.498176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.498176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.498228 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.498388 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.498389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.498389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.498524 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:23.498581 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.498745 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.498807 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:26:23.498907 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.493587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35887 (local address 127.9.90.66:49167)
0619 10:26:23.493718 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.493736 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.493807 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.493926 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.493926 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.494112 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:23.494188 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.494190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.495031 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.495032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.495582 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.495585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.495918 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.495919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.495977 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.496360 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:23.496364 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:23.496613 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:23.497963 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.497964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.497964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.498117 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.498282 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.498282 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.498282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.498339 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.498579 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:23.498580 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:23.498647 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.498713 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.498805 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for oryx/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab.
Entry for principal oryx/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab.
I20260619 10:26:23.514596 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:35887
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab
--principal=oryx/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:23.592255 17429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:23.592463 17429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:23.592485 17429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:23.593639 17429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:23.593684 17429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:23.593717 17429 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 10:26:23.593750 17429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:23.595261 17429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab
--principal=oryx/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.90.126:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:23.595439 17429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:23.595604 17429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:23.600193 17435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:23.600167 17437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:23.600180 17434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:23.600422 17429 server_base.cc:1034] running on GCE node
I20260619 10:26:23.600605 17429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:23.600826 17429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:23.601953 17429 hybrid_clock.cc:648] HybridClock initialized: now 1781864783601935 us; error 35 us; skew 500 ppm
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:23.604128 17429 init.cc:377] Logged in from keytab as oryx/127.9.90.67@KRBTEST.COM (short username oryx)
I20260619 10:26:23.604537 17429 webserver.cc:460] Webserver started at http://127.9.90.67:33159/ using document root <none> and password file <none>
I20260619 10:26:23.604732 17429 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:23.604779 17429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:23.604898 17429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:23.605906 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "332de27cfcd34451a1675a68758bda05"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "535a77f63b302914673191a5425fd8c7"
server_key_iv: "6253385a2c1eeb1614a2cf0a88b3fbab"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.606217 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "332de27cfcd34451a1675a68758bda05"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "535a77f63b302914673191a5425fd8c7"
server_key_iv: "6253385a2c1eeb1614a2cf0a88b3fbab"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.607378 17429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:23.607995 17444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.608132 17429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:23.608242 17429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "332de27cfcd34451a1675a68758bda05"
format_stamp: "Formatted at 2026-06-19 10:26:23 on dist-test-slave-hw7q"
server_key: "535a77f63b302914673191a5425fd8c7"
server_key_iv: "6253385a2c1eeb1614a2cf0a88b3fbab"
server_key_version: "encryptionkey@0"
I20260619 10:26:23.608299 17429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:23.614938 17429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:23.616423 17429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:23.616540 17429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:23.616947 17429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:23.616987 17429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.617012 17429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:23.617029 17429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.623126 17429 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:43217
I20260619 10:26:23.623185 17559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:43217 every 8 connection(s)
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:23.623461 17429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:26:23.625993 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.617345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:42431 (local address 127.9.90.126:35887)
0619 10:26:23.617409 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.617412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.617594 (+ 182us) server_negotiation.cc:407] Connection header received
0619 10:26:23.618074 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.618075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.618088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.618157 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.618896 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.619633 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.621065 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.621350 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.621417 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.623649 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:23.623651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:23.623652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:23.623660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:23.624762 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.625098 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.625100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.625100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.625153 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.625366 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.625366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.625367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.625502 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:23.625562 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.625830 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.625906 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:26:23.626139 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.617300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35887 (local address 127.9.90.67:42431)
0619 10:26:23.617499 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.617526 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.617988 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.618223 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.618226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.618533 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:23.618799 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.618805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.619720 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.619721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.620968 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.620970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.621427 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.621428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.621525 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.622013 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:23.622022 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:23.623566 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:23.624832 (+ 1266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.624837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.624839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.625035 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.625205 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.625206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.625206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.625312 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.625560 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:23.625566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:23.625692 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.625779 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.625950 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
I20260619 10:26:23.626634 17560 heartbeater.cc:347] Connected to a master server at 127.9.90.126:35887
I20260619 10:26:23.626710 17560 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:23.626883 17560 heartbeater.cc:511] Master 127.9.90.126:35887 requested a full tablet report, sending...
I20260619 10:26:23.627238 17076 ts_manager.cc:194] Registered new tserver with Master: 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217)
I20260619 10:26:23.627693 17076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:42431
I20260619 10:26:23.628535 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 17429
I20260619 10:26:23.628633 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:23.628810 9577 external_mini_cluster.cc:1428] Setting key 79705ddc111a033e4d1bbb8f6875f2ed
I20260619 10:26:23.629594 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:26:23.632468 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.627497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35887 (local address 127.9.90.67:35693)
0619 10:26:23.627602 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.627611 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.627660 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.627794 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.627794 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.627943 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:23.628002 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.628003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.628607 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.628608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.629335 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.629337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.629687 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.629688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.629766 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.630141 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:23.630144 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:23.630415 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:23.631675 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.631676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.631676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.631829 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.631985 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.631985 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.631986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.632039 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.632238 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:23.632239 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:23.632277 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.632335 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.632405 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":61,"threads_started":1}
I20260619 10:26:23.632513 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.627561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:35693 (local address 127.9.90.126:35887)
0619 10:26:23.627598 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.627600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.627650 (+ 50us) server_negotiation.cc:407] Connection header received
0619 10:26:23.627697 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.627697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.627707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.627772 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.628072 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.628523 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.629404 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.629612 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.629665 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.630493 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:23.630494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:23.630494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:23.630501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:23.631598 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.631887 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.631888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.631889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.631936 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.632088 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.632089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.632089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.632182 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:23.632252 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.632370 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.632433 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 10:26:23.649160 17144 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:23.644537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.9.90.126:35887)
0619 10:26:23.644588 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.644590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.644597 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:23.644642 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.644643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.644650 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.644695 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.645188 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.645659 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.646354 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.646537 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.646583 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.649104 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53440: BlockingRecv error: recv got EOF from 127.0.0.1:53440 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:23.655957 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.649991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.9.90.126:35887)
0619 10:26:23.650033 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.650034 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.650114 (+ 80us) server_negotiation.cc:407] Connection header received
0619 10:26:23.650155 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.650155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.650162 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.650201 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.650556 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.651011 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.651646 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.651825 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.651868 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.653767 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:23.653769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:23.653769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:23.653776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:23.654971 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.655280 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.655281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.655281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.655334 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.655488 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.655489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.655489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.655590 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:23.655673 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.655804 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.655869 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:26:23.657307 17076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:23.658846 17076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:23.664521 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.660808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:39659 (local address 127.0.0.1:42172)
0619 10:26:23.661275 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.661284 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.661341 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.661549 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.661549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.661552 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:23.661589 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.661591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.662780 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.662781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.663519 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.663522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.664280 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.664282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.664364 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.664377 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.664433 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.664471 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":51,"threads_started":1}
I20260619 10:26:23.664592 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.660910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.9.90.66:39659)
0619 10:26:23.661066 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.661070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.661351 (+ 281us) server_negotiation.cc:407] Connection header received
0619 10:26:23.661397 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.661399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.661417 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.661485 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:23.661664 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.662705 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.663590 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.664194 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.664271 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.664343 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.664460 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.664499 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260619 10:26:23.665166 17354 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf9019bef254ceb8bab5c1e023c3d05 (DEFAULT_TABLE table=test-table [id=36052072f7d94a89b0b1ae15b9196661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:23.665530 17354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf9019bef254ceb8bab5c1e023c3d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.665534 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.660629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:43217 (local address 127.0.0.1:46706)
0619 10:26:23.660784 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.660826 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.660887 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.661235 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.661238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.661240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:23.661299 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.661301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.662573 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.662574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.663854 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.663857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.665302 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.665303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.665373 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.665378 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.665430 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.665470 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":50,"thread_start_us":53,"threads_started":1}
I20260619 10:26:23.665642 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.660658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36801 (local address 127.0.0.1:38930)
0619 10:26:23.660899 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.660909 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.660960 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.661473 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.661474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.661476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:23.661524 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.661526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.662526 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.662527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.663855 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.663857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.665382 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.665383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.665446 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.665450 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.665480 (+ 30us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.665515 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":41,"thread_start_us":47,"threads_started":1}
I20260619 10:26:23.665840 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.661118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.9.90.65:36801)
0619 10:26:23.661251 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.661255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.661265 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:23.661306 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.661308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.661323 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.661425 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:23.661587 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.662450 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.663938 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.664631 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.665577 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.665664 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.665689 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.665735 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260619 10:26:23.665973 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.660926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.9.90.67:43217)
0619 10:26:23.661084 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.661088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.661098 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:23.661131 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.661132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.661144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.661211 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:23.661377 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.662470 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.664716 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.665220 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.665277 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.665322 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.665457 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.665500 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260619 10:26:23.666538 17493 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf9019bef254ceb8bab5c1e023c3d05 (DEFAULT_TABLE table=test-table [id=36052072f7d94a89b0b1ae15b9196661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:23.666862 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf9019bef254ceb8bab5c1e023c3d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.668753 17585 tablet_bootstrap.cc:492] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: Bootstrap starting.
I20260619 10:26:23.669097 17586 tablet_bootstrap.cc:492] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: Bootstrap starting.
I20260619 10:26:23.669283 17214 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf9019bef254ceb8bab5c1e023c3d05 (DEFAULT_TABLE table=test-table [id=36052072f7d94a89b0b1ae15b9196661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:23.669519 17585 tablet_bootstrap.cc:654] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.669627 17214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf9019bef254ceb8bab5c1e023c3d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.669874 17585 log.cc:824] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:23.670126 17586 tablet_bootstrap.cc:654] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.670413 17586 log.cc:824] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:23.670603 17585 tablet_bootstrap.cc:492] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: No bootstrap required, opened a new log
I20260619 10:26:23.670678 17585 ts_tablet_manager.cc:1397] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:26:23.670992 17586 tablet_bootstrap.cc:492] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: No bootstrap required, opened a new log
I20260619 10:26:23.671049 17586 ts_tablet_manager.cc:1397] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:26:23.672356 17586 raft_consensus.cc:348] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.672483 17586 raft_consensus.cc:374] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.672514 17586 raft_consensus.cc:729] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332de27cfcd34451a1675a68758bda05, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.672650 17586 consensus_queue.cc:260] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.672864 17585 raft_consensus.cc:348] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.673012 17585 raft_consensus.cc:374] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.673048 17585 raft_consensus.cc:729] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e504c6ffbe34d30a5bd82392b652b0e, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.673090 17586 ts_tablet_manager.cc:1428] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:26:23.673205 17560 heartbeater.cc:503] Master 127.9.90.126:35887 was elected leader, sending a full tablet report...
I20260619 10:26:23.673195 17585 consensus_queue.cc:260] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.673532 17420 heartbeater.cc:503] Master 127.9.90.126:35887 was elected leader, sending a full tablet report...
I20260619 10:26:23.673604 17585 ts_tablet_manager.cc:1428] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:26:23.673759 17590 tablet_bootstrap.cc:492] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: Bootstrap starting.
I20260619 10:26:23.674338 17590 tablet_bootstrap.cc:654] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.674597 17590 log.cc:824] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:23.675155 17590 tablet_bootstrap.cc:492] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: No bootstrap required, opened a new log
I20260619 10:26:23.675205 17590 ts_tablet_manager.cc:1397] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:23.676467 17590 raft_consensus.cc:348] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.676584 17590 raft_consensus.cc:374] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.676604 17590 raft_consensus.cc:729] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19e52fbd5ee4117aeb7e18f41f2018c, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.676679 17590 consensus_queue.cc:260] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.676895 17590 ts_tablet_manager.cc:1428] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:26:23.676950 17280 heartbeater.cc:503] Master 127.9.90.126:35887 was elected leader, sending a full tablet report...
W20260619 10:26:23.740612 17421 tablet.cc:2369] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:26:23.756115 17593 raft_consensus.cc:482] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:23.756245 17593 raft_consensus.cc:504] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.756563 17593 leader_election.cc:288] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217), 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659)
I20260619 10:26:23.759305 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.756794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:57285 (local address 127.9.90.67:43217)
0619 10:26:23.756851 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.756853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.756863 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:23.756906 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.756907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.756918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.756970 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:23.757163 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.757674 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.758642 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.758975 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.759026 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.759036 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.759195 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.759227 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:26:23.759305 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.756924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:37913 (local address 127.9.90.66:39659)
0619 10:26:23.756981 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.756983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.757088 (+ 105us) server_negotiation.cc:407] Connection header received
0619 10:26:23.757113 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.757114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.757126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.757173 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:23.757302 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.757758 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.758654 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.758997 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.759060 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.759070 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.759204 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.759238 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:26:23.759316 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.756734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:43217 (local address 127.9.90.65:57285)
0619 10:26:23.756793 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.756803 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.756855 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.757019 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.757019 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.757021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:23.757092 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.757095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.757744 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.757745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.758554 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.758556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.759046 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.759047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.759127 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.759130 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.759171 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.759226 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":4}
I20260619 10:26:23.759328 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.756835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:39659 (local address 127.9.90.65:37913)
0619 10:26:23.756998 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.757006 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.757051 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.757182 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.757182 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.757184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:23.757243 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.757244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.757865 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.757865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.758564 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.758566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.759075 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.759076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.759136 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.759139 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.759191 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.759227 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260619 10:26:23.759781 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf9019bef254ceb8bab5c1e023c3d05" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" is_pre_election: true
I20260619 10:26:23.759929 17374 raft_consensus.cc:2454] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 0.
I20260619 10:26:23.760138 17168 leader_election.cc:302] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e504c6ffbe34d30a5bd82392b652b0e, f19e52fbd5ee4117aeb7e18f41f2018c; no voters:
I20260619 10:26:23.760138 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf9019bef254ceb8bab5c1e023c3d05" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332de27cfcd34451a1675a68758bda05" is_pre_election: true
I20260619 10:26:23.760269 17514 raft_consensus.cc:2454] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 0.
I20260619 10:26:23.760273 17593 raft_consensus.cc:2781] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:23.760313 17593 raft_consensus.cc:482] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:23.760334 17593 raft_consensus.cc:3037] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:23.761204 17593 raft_consensus.cc:504] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.761307 17593 leader_election.cc:288] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 election: Requested vote from peers 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217), 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659)
I20260619 10:26:23.761473 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf9019bef254ceb8bab5c1e023c3d05" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332de27cfcd34451a1675a68758bda05"
I20260619 10:26:23.761482 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf9019bef254ceb8bab5c1e023c3d05" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e504c6ffbe34d30a5bd82392b652b0e"
I20260619 10:26:23.761541 17514 raft_consensus.cc:3037] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:23.761545 17374 raft_consensus.cc:3037] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:23.762236 17514 raft_consensus.cc:2454] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 1.
I20260619 10:26:23.762272 17374 raft_consensus.cc:2454] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 1.
I20260619 10:26:23.762382 17168 leader_election.cc:302] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 332de27cfcd34451a1675a68758bda05, f19e52fbd5ee4117aeb7e18f41f2018c; no voters:
I20260619 10:26:23.762462 17593 raft_consensus.cc:2781] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:23.762632 17593 raft_consensus.cc:686] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 LEADER]: Becoming Leader. State: Replica: f19e52fbd5ee4117aeb7e18f41f2018c, State: Running, Role: LEADER
I20260619 10:26:23.762717 17593 consensus_queue.cc:237] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.763525 17077 catalog_manager.cc:5495] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c reported cstate change: term changed from 0 to 1, leader changed from <none> to f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65). New cstate: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } health_report { overall_health: HEALTHY } } }
I20260619 10:26:23.779165 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.777030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38946 (local address 127.9.90.65:36801)
0619 10:26:23.777077 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.777079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.777085 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:26:23.777171 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.777172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.777188 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.777242 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:23.777342 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.777818 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.778547 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.778721 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.778772 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.778903 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:23.778996 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:23.779046 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.779092 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.779107 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:26:23.780803 17374 raft_consensus.cc:1261] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Refusing update from remote peer f19e52fbd5ee4117aeb7e18f41f2018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:23.780803 17514 raft_consensus.cc:1261] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Refusing update from remote peer f19e52fbd5ee4117aeb7e18f41f2018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:23.781124 17595 consensus_queue.cc:1035] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:23.781263 17593 consensus_queue.cc:1035] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:23.783541 17599 mvcc.cc:205] Tried to move back new op lower bound from 7298518154363654144 to 7298518154292666368. Current Snapshot: MvccSnapshot[applied={T|T < 7298518154363654144}]
I20260619 10:26:23.783807 17600 mvcc.cc:205] Tried to move back new op lower bound from 7298518154363654144 to 7298518154292666368. Current Snapshot: MvccSnapshot[applied={T|T < 7298518154363654144}]
I20260619 10:26:23.785764 17076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:23.785873 17076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:23.786576 17076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88: Sending DeleteTablet for 3 replicas of tablet 2bf9019bef254ceb8bab5c1e023c3d05
I20260619 10:26:23.786890 17214 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf9019bef254ceb8bab5c1e023c3d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:23 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:38930
I20260619 10:26:23.786957 17493 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf9019bef254ceb8bab5c1e023c3d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:23 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46706
I20260619 10:26:23.787168 17608 tablet_replica.cc:330] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: stopping tablet replica
I20260619 10:26:23.787211 17354 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf9019bef254ceb8bab5c1e023c3d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:23 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:42172
I20260619 10:26:23.787271 17608 raft_consensus.cc:2229] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:23.787439 17609 tablet_replica.cc:330] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: stopping tablet replica
I20260619 10:26:23.787572 17609 raft_consensus.cc:2229] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:23.787587 17608 raft_consensus.cc:2258] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:23.787607 17076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:23.787750 17076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:23.787905 17609 raft_consensus.cc:2258] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:23.788024 17608 ts_tablet_manager.cc:1905] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:23.788408 17609 ts_tablet_manager.cc:1905] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:23.789004 17493 tablet_service.cc:1467] Processing CreateTablet for tablet fef8bfa61d2a49f4904b137a45be31df (DEFAULT_TABLE table=test-table [id=4399d7ddd23b48298fa806a219ad8ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:23.789124 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef8bfa61d2a49f4904b137a45be31df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.789499 17608 ts_tablet_manager.cc:1918] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:23.789572 17608 log.cc:1196] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/2bf9019bef254ceb8bab5c1e023c3d05
I20260619 10:26:23.789850 17608 ts_tablet_manager.cc:1939] T 2bf9019bef254ceb8bab5c1e023c3d05 P 332de27cfcd34451a1675a68758bda05: Deleting consensus metadata
I20260619 10:26:23.790251 17609 ts_tablet_manager.cc:1918] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:23.790311 17609 log.cc:1196] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/2bf9019bef254ceb8bab5c1e023c3d05
I20260619 10:26:23.790341 17586 tablet_bootstrap.cc:492] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: Bootstrap starting.
I20260619 10:26:23.790421 17064 catalog_manager.cc:4841] TS 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217): tablet 2bf9019bef254ceb8bab5c1e023c3d05 (table test-table [id=36052072f7d94a89b0b1ae15b9196661]) successfully deleted
I20260619 10:26:23.790530 17609 ts_tablet_manager.cc:1939] T 2bf9019bef254ceb8bab5c1e023c3d05 P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting consensus metadata
I20260619 10:26:23.790726 17586 tablet_bootstrap.cc:654] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.790750 17214 tablet_service.cc:1467] Processing CreateTablet for tablet fef8bfa61d2a49f4904b137a45be31df (DEFAULT_TABLE table=test-table [id=4399d7ddd23b48298fa806a219ad8ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:23.790851 17214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef8bfa61d2a49f4904b137a45be31df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.790894 17064 catalog_manager.cc:4841] TS 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659): tablet 2bf9019bef254ceb8bab5c1e023c3d05 (table test-table [id=36052072f7d94a89b0b1ae15b9196661]) successfully deleted
I20260619 10:26:23.791280 17586 tablet_bootstrap.cc:492] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: No bootstrap required, opened a new log
I20260619 10:26:23.791319 17586 ts_tablet_manager.cc:1397] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:23.791450 17586 raft_consensus.cc:348] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:23.791502 17586 raft_consensus.cc:374] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.791520 17586 raft_consensus.cc:729] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332de27cfcd34451a1675a68758bda05, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.791558 17586 consensus_queue.cc:260] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:23.791637 17586 ts_tablet_manager.cc:1428] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.792068 17590 tablet_bootstrap.cc:492] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: Bootstrap starting.
I20260619 10:26:23.792294 17354 tablet_service.cc:1467] Processing CreateTablet for tablet fef8bfa61d2a49f4904b137a45be31df (DEFAULT_TABLE table=test-table [id=4399d7ddd23b48298fa806a219ad8ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:23.792390 17354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef8bfa61d2a49f4904b137a45be31df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.792472 17590 tablet_bootstrap.cc:654] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.793041 17590 tablet_bootstrap.cc:492] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: No bootstrap required, opened a new log
I20260619 10:26:23.793087 17590 ts_tablet_manager.cc:1397] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:23.793216 17590 raft_consensus.cc:348] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:23.793277 17590 raft_consensus.cc:374] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.793294 17590 raft_consensus.cc:729] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19e52fbd5ee4117aeb7e18f41f2018c, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.793337 17590 consensus_queue.cc:260] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:23.793422 17590 ts_tablet_manager.cc:1428] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.793534 17585 tablet_bootstrap.cc:492] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: Bootstrap starting.
I20260619 10:26:23.793923 17585 tablet_bootstrap.cc:654] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.794445 17607 tablet_replica.cc:330] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: stopping tablet replica
I20260619 10:26:23.794552 17607 raft_consensus.cc:2229] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:23.794627 17585 tablet_bootstrap.cc:492] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: No bootstrap required, opened a new log
I20260619 10:26:23.794672 17585 ts_tablet_manager.cc:1397] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:23.794680 17607 raft_consensus.cc:2258] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:23.794818 17585 raft_consensus.cc:348] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:23.794885 17585 raft_consensus.cc:374] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.794903 17585 raft_consensus.cc:729] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e504c6ffbe34d30a5bd82392b652b0e, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.794948 17585 consensus_queue.cc:260] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:23.795032 17585 ts_tablet_manager.cc:1428] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.795282 17607 ts_tablet_manager.cc:1905] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:23.796723 17607 ts_tablet_manager.cc:1918] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:23.796795 17607 log.cc:1196] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/2bf9019bef254ceb8bab5c1e023c3d05
I20260619 10:26:23.797014 17607 ts_tablet_manager.cc:1939] T 2bf9019bef254ceb8bab5c1e023c3d05 P f19e52fbd5ee4117aeb7e18f41f2018c: Deleting consensus metadata
I20260619 10:26:23.797394 17062 catalog_manager.cc:4841] TS f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65:36801): tablet 2bf9019bef254ceb8bab5c1e023c3d05 (table test-table [id=36052072f7d94a89b0b1ae15b9196661]) successfully deleted
I20260619 10:26:23.824369 17591 raft_consensus.cc:482] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:23.824460 17591 raft_consensus.cc:504] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:23.824791 17591 leader_election.cc:288] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65:36801), 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217)
I20260619 10:26:23.827818 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.825201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:41131 (local address 127.9.90.65:36801)
0619 10:26:23.825256 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.825258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.825266 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:23.825292 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.825293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.825302 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.825352 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:23.825512 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.826085 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.827094 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.827527 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.827587 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.827598 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.827718 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.827750 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:26:23.827837 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.824932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36801 (local address 127.9.90.66:41131)
0619 10:26:23.825052 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.825063 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.825116 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.825364 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.825366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.825368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:23.825437 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.825439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.826156 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.826157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.827014 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.827017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.827586 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.827586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.827659 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.827663 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.827709 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.827754 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 10:26:23.828277 17281 tablet.cc:2369] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:26:23.828279 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef8bfa61d2a49f4904b137a45be31df" candidate_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" is_pre_election: true
I20260619 10:26:23.828413 17234 raft_consensus.cc:2454] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e504c6ffbe34d30a5bd82392b652b0e in term 0.
I20260619 10:26:23.828567 17306 leader_election.cc:302] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e504c6ffbe34d30a5bd82392b652b0e, f19e52fbd5ee4117aeb7e18f41f2018c; no voters:
I20260619 10:26:23.828691 17591 raft_consensus.cc:2781] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:23.828722 17591 raft_consensus.cc:482] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:23.828737 17591 raft_consensus.cc:3037] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:23.829121 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.824943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:43217 (local address 127.9.90.66:44403)
0619 10:26:23.825257 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.825266 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.825314 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.825470 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.825471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.825473 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:23.825559 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.825561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.826545 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.826546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.827763 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.827766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.828865 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.828866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.828941 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.828945 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.828992 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.829029 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":66,"threads_started":1}
I20260619 10:26:23.829130 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.825009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:44403 (local address 127.9.90.67:43217)
0619 10:26:23.825057 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.825059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.825306 (+ 247us) server_negotiation.cc:407] Connection header received
0619 10:26:23.825390 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.825391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.825399 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.825424 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:23.825626 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.826422 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.827857 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.828427 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.828472 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.828481 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.829027 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.829069 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:23.829527 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef8bfa61d2a49f4904b137a45be31df" candidate_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332de27cfcd34451a1675a68758bda05" is_pre_election: true
I20260619 10:26:23.829559 17591 raft_consensus.cc:504] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:23.829617 17514 raft_consensus.cc:2454] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e504c6ffbe34d30a5bd82392b652b0e in term 0.
I20260619 10:26:23.829663 17591 leader_election.cc:288] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [CANDIDATE]: Term 1 election: Requested vote from peers f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65:36801), 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217)
I20260619 10:26:23.829846 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef8bfa61d2a49f4904b137a45be31df" candidate_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332de27cfcd34451a1675a68758bda05"
I20260619 10:26:23.829905 17514 raft_consensus.cc:3037] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:23.829998 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef8bfa61d2a49f4904b137a45be31df" candidate_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c"
I20260619 10:26:23.830082 17234 raft_consensus.cc:3037] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:23.830622 17514 raft_consensus.cc:2454] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e504c6ffbe34d30a5bd82392b652b0e in term 1.
I20260619 10:26:23.830785 17234 raft_consensus.cc:2454] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e504c6ffbe34d30a5bd82392b652b0e in term 1.
I20260619 10:26:23.830812 17308 leader_election.cc:302] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 332de27cfcd34451a1675a68758bda05, 6e504c6ffbe34d30a5bd82392b652b0e; no voters:
I20260619 10:26:23.830878 17591 raft_consensus.cc:2781] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:23.831000 17591 raft_consensus.cc:686] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 LEADER]: Becoming Leader. State: Replica: 6e504c6ffbe34d30a5bd82392b652b0e, State: Running, Role: LEADER
I20260619 10:26:23.831067 17591 consensus_queue.cc:237] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:23.831657 17077 catalog_manager.cc:5495] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66). New cstate: current_term: 1 leader_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } health_report { overall_health: UNKNOWN } } }
Jun 19 10:26:23 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.126@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:23.864805 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.858424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35887 (local address 127.0.0.1:53456)
0619 10:26:23.858562 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:23.858570 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:23.858629 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:23.858767 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:23.858768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:23.859040 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:23.859082 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.859084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.859707 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.859708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.860208 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:23.860210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.860506 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.860507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:23.860554 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.861008 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:23.861012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:23.862506 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:23.863814 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.863820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.863822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.863992 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.864198 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:23.864199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:23.864199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:23.864262 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:23.864460 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:23.864466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:23.864509 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 10:26:23.864567 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:26:23.864687 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260619 10:26:23.864845 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:23.858456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.9.90.126:35887)
0619 10:26:23.858500 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:23.858502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:23.858622 (+ 120us) server_negotiation.cc:407] Connection header received
0619 10:26:23.858684 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:23.858685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:23.858694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:23.858762 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:23.859151 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.859621 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.860279 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:23.860454 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:23.860504 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:23.862586 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:23.862587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:23.862587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:23.862593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:23.863745 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.864051 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.864052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.864053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.864123 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:23.864310 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:23.864311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:23.864312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:23.864408 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:23.864461 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:23.864593 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 10:26:23.864739 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":24}
I20260619 10:26:23.866932 17077 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:53456:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 10:26:23.867122 17077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:23.868475 17493 tablet_service.cc:1467] Processing CreateTablet for tablet 689d925d310c40dbb2c162d11590d64e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51d3ee32ec9e485db842b9b599133467]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:26:23.868462 17354 tablet_service.cc:1467] Processing CreateTablet for tablet 689d925d310c40dbb2c162d11590d64e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51d3ee32ec9e485db842b9b599133467]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:26:23.868598 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 689d925d310c40dbb2c162d11590d64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.868611 17354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 689d925d310c40dbb2c162d11590d64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.869606 17586 tablet_bootstrap.cc:492] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: Bootstrap starting.
I20260619 10:26:23.869973 17586 tablet_bootstrap.cc:654] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.870141 17585 tablet_bootstrap.cc:492] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: Bootstrap starting.
I20260619 10:26:23.870272 17214 tablet_service.cc:1467] Processing CreateTablet for tablet 689d925d310c40dbb2c162d11590d64e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51d3ee32ec9e485db842b9b599133467]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 10:26:23.870406 17214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 689d925d310c40dbb2c162d11590d64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:23.870507 17585 tablet_bootstrap.cc:654] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.870674 17586 tablet_bootstrap.cc:492] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: No bootstrap required, opened a new log
I20260619 10:26:23.870721 17586 ts_tablet_manager.cc:1397] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:23.870857 17586 raft_consensus.cc:348] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.870931 17586 raft_consensus.cc:374] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.870950 17586 raft_consensus.cc:729] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332de27cfcd34451a1675a68758bda05, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.870990 17586 consensus_queue.cc:260] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.871073 17586 ts_tablet_manager.cc:1428] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.871169 17585 tablet_bootstrap.cc:492] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: No bootstrap required, opened a new log
I20260619 10:26:23.871202 17585 ts_tablet_manager.cc:1397] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:23.871212 17589 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:23.871299 17589 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:23.871493 17585 raft_consensus.cc:348] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.871541 17585 raft_consensus.cc:374] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.871552 17585 raft_consensus.cc:729] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e504c6ffbe34d30a5bd82392b652b0e, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.871580 17585 consensus_queue.cc:260] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.871647 17585 ts_tablet_manager.cc:1428] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.871702 17591 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:23.871759 17591 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:23.871928 17590 tablet_bootstrap.cc:492] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: Bootstrap starting.
I20260619 10:26:23.872313 17590 tablet_bootstrap.cc:654] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:23.872885 17590 tablet_bootstrap.cc:492] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: No bootstrap required, opened a new log
I20260619 10:26:23.872942 17590 ts_tablet_manager.cc:1397] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:23.873044 17590 raft_consensus.cc:348] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.873102 17590 raft_consensus.cc:374] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:23.873118 17590 raft_consensus.cc:729] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19e52fbd5ee4117aeb7e18f41f2018c, State: Initialized, Role: FOLLOWER
I20260619 10:26:23.873157 17590 consensus_queue.cc:260] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:23.873226 17590 ts_tablet_manager.cc:1428] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:23.873317 17595 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:23.873375 17595 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 10:26:23.873886 17561 tablet.cc:2369] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:26:24.062474 17595 raft_consensus.cc:482] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:24.062587 17595 raft_consensus.cc:504] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:24.062733 17595 leader_election.cc:288] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659), 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217)
I20260619 10:26:24.062973 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689d925d310c40dbb2c162d11590d64e" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" is_pre_election: true
I20260619 10:26:24.062999 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689d925d310c40dbb2c162d11590d64e" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332de27cfcd34451a1675a68758bda05" is_pre_election: true
I20260619 10:26:24.063088 17514 raft_consensus.cc:2454] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 0.
I20260619 10:26:24.063088 17374 raft_consensus.cc:2454] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 0.
I20260619 10:26:24.063246 17168 leader_election.cc:302] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 332de27cfcd34451a1675a68758bda05, f19e52fbd5ee4117aeb7e18f41f2018c; no voters:
I20260619 10:26:24.063349 17595 raft_consensus.cc:2781] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:24.063388 17595 raft_consensus.cc:482] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:24.063406 17595 raft_consensus.cc:3037] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:24.064090 17595 raft_consensus.cc:504] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:24.064193 17595 leader_election.cc:288] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 election: Requested vote from peers 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659), 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217)
I20260619 10:26:24.064383 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689d925d310c40dbb2c162d11590d64e" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e504c6ffbe34d30a5bd82392b652b0e"
I20260619 10:26:24.064389 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689d925d310c40dbb2c162d11590d64e" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332de27cfcd34451a1675a68758bda05"
I20260619 10:26:24.064461 17514 raft_consensus.cc:3037] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:24.064461 17374 raft_consensus.cc:3037] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:24.065173 17374 raft_consensus.cc:2454] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 1.
I20260619 10:26:24.065173 17514 raft_consensus.cc:2454] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 1.
I20260619 10:26:24.065322 17168 leader_election.cc:302] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 332de27cfcd34451a1675a68758bda05, f19e52fbd5ee4117aeb7e18f41f2018c; no voters:
I20260619 10:26:24.065413 17595 raft_consensus.cc:2781] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:24.065464 17595 raft_consensus.cc:686] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 LEADER]: Becoming Leader. State: Replica: f19e52fbd5ee4117aeb7e18f41f2018c, State: Running, Role: LEADER
I20260619 10:26:24.065510 17595 consensus_queue.cc:237] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } }
I20260619 10:26:24.065757 17593 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: TxnStatusTablet state changed. Reason: New leader f19e52fbd5ee4117aeb7e18f41f2018c. Latest consensus state: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:24.065865 17593 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:26:24.066004 17626 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 10:26:24.066128 17626 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 10:26:24.066103 17077 catalog_manager.cc:5495] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c reported cstate change: term changed from 0 to 1, leader changed from <none> to f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65). New cstate: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } health_report { overall_health: HEALTHY } } }
I20260619 10:26:24.101035 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.098823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.9.90.65:36801)
0619 10:26:24.098873 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.098875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.098882 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:24.098905 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.098906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.098918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.098977 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:24.099108 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.099572 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.100384 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.100581 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.100633 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.100798 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.100834 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.100879 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.100936 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.100956 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:26:24.101047 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.098578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36801 (local address 127.0.0.1:38960)
0619 10:26:24.098673 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:24.098686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:24.098738 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:24.098996 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:24.098998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:24.098999 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:24.099050 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.099053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.099640 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.099641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.100307 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.100310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.100637 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.100639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.100708 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.100727 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.100885 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.100891 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 10:26:24.100932 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 10:26:24.100982 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 10:26:24.101933 17514 raft_consensus.cc:1261] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Refusing update from remote peer f19e52fbd5ee4117aeb7e18f41f2018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:24.101972 17374 raft_consensus.cc:1261] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Refusing update from remote peer f19e52fbd5ee4117aeb7e18f41f2018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:24.102176 17595 consensus_queue.cc:1035] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:24.102300 17593 consensus_queue.cc:1035] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:24.102720 17591 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: TxnStatusTablet state changed. Reason: New leader f19e52fbd5ee4117aeb7e18f41f2018c. Latest consensus state: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:24.102757 17589 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: TxnStatusTablet state changed. Reason: New leader f19e52fbd5ee4117aeb7e18f41f2018c. Latest consensus state: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:24.102819 17591 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:24.102842 17589 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:24.103246 17593 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:24.103322 17593 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:26:24.103363 17598 mvcc.cc:205] Tried to move back new op lower bound from 7298518155679870976 to 7298518155532673024. Current Snapshot: MvccSnapshot[applied={T|T < 7298518155679870976 or (T in {7298518155679870976})}]
I20260619 10:26:24.103760 17589 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:24.103837 17589 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P 332de27cfcd34451a1675a68758bda05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:24.104481 17595 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:24.104554 17595 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P f19e52fbd5ee4117aeb7e18f41f2018c: This TxnStatusTablet replica's current role is: LEADER
I20260619 10:26:24.104794 17591 tablet_replica.cc:439] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } } peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } }
I20260619 10:26:24.104873 17591 tablet_replica.cc:442] T 689d925d310c40dbb2c162d11590d64e P 6e504c6ffbe34d30a5bd82392b652b0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 10:26:24.107290 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.105128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.9.90.66:39659)
0619 10:26:24.105161 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.105163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.105169 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:26:24.105199 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.105200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.105218 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.105273 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:24.105418 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.105934 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.106602 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.106805 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.106858 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.106991 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.107094 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.107153 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.107200 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.107226 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:26:24.111632 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.109498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:46041 (local address 127.9.90.65:36801)
0619 10:26:24.109537 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.109538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.109562 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:26:24.109616 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.109617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.109626 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.109674 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:24.109810 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.110248 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.111043 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.111238 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.111288 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.111428 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.111461 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.111500 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.111547 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.111565 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:26:24.111658 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.109370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36801 (local address 127.9.90.66:46041)
0619 10:26:24.109508 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:24.109519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:24.109574 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:24.109677 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:24.109678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:24.109679 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:24.109739 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.109740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.110311 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.110312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.110974 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.110976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.111293 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.111295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.111351 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.111363 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.111506 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.111507 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:26:24.111546 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 10:26:24.111585 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260619 10:26:24.113510 17514 raft_consensus.cc:1261] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Refusing update from remote peer 6e504c6ffbe34d30a5bd82392b652b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:24.113673 17234 raft_consensus.cc:1261] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Refusing update from remote peer 6e504c6ffbe34d30a5bd82392b652b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:24.113761 17591 consensus_queue.cc:1035] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:24.113907 17614 consensus_queue.cc:1035] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:24.122295 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.119640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:39659 (local address 127.9.90.65:36649)
0619 10:26:24.119790 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:24.119800 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:24.119875 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:24.120016 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:24.120017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:24.120019 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:24.120065 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.120067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.120849 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.120850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.121447 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.121449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.121921 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.121922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.121960 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.121973 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.122139 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.122140 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:26:24.122171 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 10:26:24.122216 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260619 10:26:24.122293 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.119693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:36649 (local address 127.9.90.66:39659)
0619 10:26:24.119727 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.119728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.119849 (+ 121us) server_negotiation.cc:407] Connection header received
0619 10:26:24.119944 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.119945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.119956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.120013 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:24.120120 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.120775 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.121527 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.121854 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.121926 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.122045 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.122080 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.122140 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.122186 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.122205 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:26:24.129905 17077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:24.130019 17077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:24.130684 17077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88: Sending DeleteTablet for 3 replicas of tablet fef8bfa61d2a49f4904b137a45be31df
I20260619 10:26:24.130920 17354 tablet_service.cc:1514] Processing DeleteTablet for tablet fef8bfa61d2a49f4904b137a45be31df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:24 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:42172
I20260619 10:26:24.131003 17609 tablet_replica.cc:330] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: stopping tablet replica
I20260619 10:26:24.131013 17214 tablet_service.cc:1514] Processing DeleteTablet for tablet fef8bfa61d2a49f4904b137a45be31df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:24 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:38930
I20260619 10:26:24.131042 17609 raft_consensus.cc:2229] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:24.131098 17607 tablet_replica.cc:330] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: stopping tablet replica
I20260619 10:26:24.131116 17609 raft_consensus.cc:2258] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:24.131140 17607 raft_consensus.cc:2229] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:24.131170 17607 raft_consensus.cc:2258] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:24.131338 17493 tablet_service.cc:1514] Processing DeleteTablet for tablet fef8bfa61d2a49f4904b137a45be31df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:24 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46706
I20260619 10:26:24.131420 17608 tablet_replica.cc:330] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: stopping tablet replica
I20260619 10:26:24.131462 17608 raft_consensus.cc:2229] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:24.131492 17608 raft_consensus.cc:2258] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:24.131616 17609 ts_tablet_manager.cc:1905] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:24.131887 17608 ts_tablet_manager.cc:1905] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:24.133183 17609 ts_tablet_manager.cc:1918] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:24.133241 17609 log.cc:1196] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/fef8bfa61d2a49f4904b137a45be31df
I20260619 10:26:24.133419 17608 ts_tablet_manager.cc:1918] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:24.133437 17609 ts_tablet_manager.cc:1939] T fef8bfa61d2a49f4904b137a45be31df P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting consensus metadata
I20260619 10:26:24.133471 17608 log.cc:1196] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/fef8bfa61d2a49f4904b137a45be31df
I20260619 10:26:24.133661 17608 ts_tablet_manager.cc:1939] T fef8bfa61d2a49f4904b137a45be31df P 332de27cfcd34451a1675a68758bda05: Deleting consensus metadata
I20260619 10:26:24.134065 17607 ts_tablet_manager.cc:1905] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:24.134063 17064 catalog_manager.cc:4841] TS 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217): tablet fef8bfa61d2a49f4904b137a45be31df (table test-table [id=4399d7ddd23b48298fa806a219ad8ed3]) successfully deleted
I20260619 10:26:24.135468 17607 ts_tablet_manager.cc:1918] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:24.135512 17607 log.cc:1196] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/fef8bfa61d2a49f4904b137a45be31df
I20260619 10:26:24.135689 17607 ts_tablet_manager.cc:1939] T fef8bfa61d2a49f4904b137a45be31df P f19e52fbd5ee4117aeb7e18f41f2018c: Deleting consensus metadata
I20260619 10:26:24.136000 17062 catalog_manager.cc:4841] TS f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65:36801): tablet fef8bfa61d2a49f4904b137a45be31df (table test-table [id=4399d7ddd23b48298fa806a219ad8ed3]) successfully deleted
I20260619 10:26:24.136240 17064 catalog_manager.cc:4841] TS 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659): tablet fef8bfa61d2a49f4904b137a45be31df (table test-table [id=4399d7ddd23b48298fa806a219ad8ed3]) successfully deleted
Jun 19 10:26:24 dist-test-slave-hw7q krb5kdc[17032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 10:26:24 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:24 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 10:26:24.167441 17144 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:24.160659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.9.90.126:35887)
0619 10:26:24.160705 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.160707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.160852 (+ 145us) server_negotiation.cc:407] Connection header received
0619 10:26:24.160924 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.160925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.160937 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.161003 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:24.161684 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.162568 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.163572 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.163870 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.163936 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.167374 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53468: BlockingRecv error: recv got EOF from 127.0.0.1:53468 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 10:26:24 dist-test-slave-hw7q krb5kdc[17032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM
I20260619 10:26:24.182266 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.174046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.9.90.126:35887)
0619 10:26:24.174092 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.174094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.174147 (+ 53us) server_negotiation.cc:407] Connection header received
0619 10:26:24.174230 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.174230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.174241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.174293 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:24.174857 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.175599 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.176614 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.176913 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.176981 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.179555 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:24.179556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:24.179557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:24.179564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:24.181120 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:24.181489 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:24.181490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:24.181490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:24.181547 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:24.181716 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:24.181716 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:24.181717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:24.181877 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:24.181965 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.182088 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.182169 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:26:24.183804 17077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:24.183967 17077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:24.185545 17214 tablet_service.cc:1467] Processing CreateTablet for tablet 1d459d3bac6c4a048d7a594d444579a7 (DEFAULT_TABLE table=test-table [id=d634bc67ccf1473fa69ff3b2e781fb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:24.185546 17354 tablet_service.cc:1467] Processing CreateTablet for tablet 1d459d3bac6c4a048d7a594d444579a7 (DEFAULT_TABLE table=test-table [id=d634bc67ccf1473fa69ff3b2e781fb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:24.185683 17214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d459d3bac6c4a048d7a594d444579a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:24.185683 17354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d459d3bac6c4a048d7a594d444579a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:24.186996 17493 tablet_service.cc:1467] Processing CreateTablet for tablet 1d459d3bac6c4a048d7a594d444579a7 (DEFAULT_TABLE table=test-table [id=d634bc67ccf1473fa69ff3b2e781fb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:24.187110 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d459d3bac6c4a048d7a594d444579a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:24.188350 17590 tablet_bootstrap.cc:492] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: Bootstrap starting.
I20260619 10:26:24.188627 17590 tablet_bootstrap.cc:654] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:24.188858 17585 tablet_bootstrap.cc:492] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: Bootstrap starting.
I20260619 10:26:24.189179 17585 tablet_bootstrap.cc:654] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:24.189705 17586 tablet_bootstrap.cc:492] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: Bootstrap starting.
I20260619 10:26:24.190092 17586 tablet_bootstrap.cc:654] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:24.191112 17590 tablet_bootstrap.cc:492] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: No bootstrap required, opened a new log
I20260619 10:26:24.191159 17590 ts_tablet_manager.cc:1397] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 10:26:24.191273 17590 raft_consensus.cc:348] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.191337 17590 raft_consensus.cc:374] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:24.191355 17590 raft_consensus.cc:729] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19e52fbd5ee4117aeb7e18f41f2018c, State: Initialized, Role: FOLLOWER
I20260619 10:26:24.191401 17590 consensus_queue.cc:260] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.191505 17590 ts_tablet_manager.cc:1428] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:24.192046 17586 tablet_bootstrap.cc:492] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: No bootstrap required, opened a new log
I20260619 10:26:24.192091 17586 ts_tablet_manager.cc:1397] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:24.192268 17586 raft_consensus.cc:348] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.192327 17586 raft_consensus.cc:374] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:24.192343 17586 raft_consensus.cc:729] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332de27cfcd34451a1675a68758bda05, State: Initialized, Role: FOLLOWER
I20260619 10:26:24.192378 17586 consensus_queue.cc:260] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.192459 17586 ts_tablet_manager.cc:1428] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:24.193207 17585 tablet_bootstrap.cc:492] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: No bootstrap required, opened a new log
I20260619 10:26:24.193270 17585 ts_tablet_manager.cc:1397] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 10:26:24.193392 17585 raft_consensus.cc:348] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.193456 17585 raft_consensus.cc:374] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:24.193480 17585 raft_consensus.cc:729] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e504c6ffbe34d30a5bd82392b652b0e, State: Initialized, Role: FOLLOWER
I20260619 10:26:24.193522 17585 consensus_queue.cc:260] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.193616 17585 ts_tablet_manager.cc:1428] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:24.233001 17589 raft_consensus.cc:482] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:24.233096 17589 raft_consensus.cc:504] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.233405 17589 leader_election.cc:288] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65:36801), 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659)
I20260619 10:26:24.236488 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.233715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:46019 (local address 127.9.90.66:39659)
0619 10:26:24.233768 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.233769 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.233862 (+ 93us) server_negotiation.cc:407] Connection header received
0619 10:26:24.233934 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.233935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.233944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.233990 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:24.234125 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.234869 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.235837 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.236184 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.236240 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.236251 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.236391 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.236423 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:26:24.236497 17658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.233569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:39659 (local address 127.9.90.67:46019)
0619 10:26:24.233808 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:24.233816 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:24.233883 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:24.234002 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:24.234003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:24.234004 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:24.234072 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.234074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.234990 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.234991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.235751 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.235753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.236275 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.236276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.236331 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.236334 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:24.236378 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 10:26:24.236417 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":74,"threads_started":1}
I20260619 10:26:24.237080 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d459d3bac6c4a048d7a594d444579a7" candidate_uuid: "332de27cfcd34451a1675a68758bda05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" is_pre_election: true
I20260619 10:26:24.237182 17374 raft_consensus.cc:2454] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 332de27cfcd34451a1675a68758bda05 in term 0.
I20260619 10:26:24.237392 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.233662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:55993 (local address 127.9.90.65:36801)
0619 10:26:24.233730 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.233732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.233826 (+ 94us) server_negotiation.cc:407] Connection header received
0619 10:26:24.233889 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.233890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.233900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.233956 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:24.234189 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.234874 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.236520 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.237023 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.237075 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.237085 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.237276 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.237329 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 10:26:24.237460 17448 leader_election.cc:302] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 332de27cfcd34451a1675a68758bda05, 6e504c6ffbe34d30a5bd82392b652b0e; no voters:
I20260619 10:26:24.237568 17589 raft_consensus.cc:2781] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:24.237609 17589 raft_consensus.cc:482] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:24.237634 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.233552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36801 (local address 127.9.90.67:55993)
0619 10:26:24.233727 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:24.233738 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:24.233827 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:24.234007 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:24.234008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:24.234011 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:24.234098 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.234100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.234950 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.234951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.236416 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.236419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.237093 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.237094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.237171 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.237175 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:24.237495 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 10:26:24.237531 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":15,"thread_start_us":53,"threads_started":1}
I20260619 10:26:24.237627 17589 raft_consensus.cc:3037] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:24.237869 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d459d3bac6c4a048d7a594d444579a7" candidate_uuid: "332de27cfcd34451a1675a68758bda05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" is_pre_election: true
I20260619 10:26:24.237967 17234 raft_consensus.cc:2454] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 332de27cfcd34451a1675a68758bda05 in term 0.
I20260619 10:26:24.238461 17589 raft_consensus.cc:504] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.238590 17589 leader_election.cc:288] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [CANDIDATE]: Term 1 election: Requested vote from peers f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65:36801), 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659)
I20260619 10:26:24.238725 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d459d3bac6c4a048d7a594d444579a7" candidate_uuid: "332de27cfcd34451a1675a68758bda05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e504c6ffbe34d30a5bd82392b652b0e"
I20260619 10:26:24.238785 17374 raft_consensus.cc:3037] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:24.238737 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d459d3bac6c4a048d7a594d444579a7" candidate_uuid: "332de27cfcd34451a1675a68758bda05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c"
I20260619 10:26:24.238811 17234 raft_consensus.cc:3037] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:24.239326 17374 raft_consensus.cc:2454] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 332de27cfcd34451a1675a68758bda05 in term 1.
I20260619 10:26:24.239457 17448 leader_election.cc:302] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 332de27cfcd34451a1675a68758bda05, 6e504c6ffbe34d30a5bd82392b652b0e; no voters:
I20260619 10:26:24.239535 17589 raft_consensus.cc:2781] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:24.239629 17589 raft_consensus.cc:686] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 1 LEADER]: Becoming Leader. State: Replica: 332de27cfcd34451a1675a68758bda05, State: Running, Role: LEADER
I20260619 10:26:24.239670 17234 raft_consensus.cc:2454] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 332de27cfcd34451a1675a68758bda05 in term 1.
I20260619 10:26:24.239686 17589 consensus_queue.cc:237] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.240339 17077 catalog_manager.cc:5495] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 332de27cfcd34451a1675a68758bda05 (127.9.90.67). New cstate: current_term: 1 leader_uuid: "332de27cfcd34451a1675a68758bda05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } health_report { overall_health: HEALTHY } } }
I20260619 10:26:24.253417 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.250990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.9.90.67:43217)
0619 10:26:24.251026 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.251028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.251050 (+ 22us) server_negotiation.cc:407] Connection header received
0619 10:26:24.251123 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.251124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.251140 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.251190 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:24.251320 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.251801 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.252828 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.252996 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.253034 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.253168 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.253256 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.253301 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.253345 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.253359 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:26:24.254347 17374 raft_consensus.cc:1261] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Refusing update from remote peer 332de27cfcd34451a1675a68758bda05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:24.254344 17234 raft_consensus.cc:1261] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Refusing update from remote peer 332de27cfcd34451a1675a68758bda05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:24.254578 17589 consensus_queue.cc:1035] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:24.254700 17659 consensus_queue.cc:1035] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:24.257860 17077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:24.257954 17077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:24.258540 17077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88: Sending DeleteTablet for 3 replicas of tablet 1d459d3bac6c4a048d7a594d444579a7
I20260619 10:26:24.258783 17214 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d459d3bac6c4a048d7a594d444579a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:24 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:38930
I20260619 10:26:24.258875 17607 tablet_replica.cc:330] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: stopping tablet replica
I20260619 10:26:24.258870 17354 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d459d3bac6c4a048d7a594d444579a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:24 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:42172
I20260619 10:26:24.258922 17607 raft_consensus.cc:2229] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:24.258958 17607 raft_consensus.cc:2258] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:24.258967 17609 tablet_replica.cc:330] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: stopping tablet replica
I20260619 10:26:24.259013 17609 raft_consensus.cc:2229] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:24.259008 17493 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d459d3bac6c4a048d7a594d444579a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:24 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46706
I20260619 10:26:24.259049 17609 raft_consensus.cc:2258] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:24.259079 17608 tablet_replica.cc:330] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: stopping tablet replica
I20260619 10:26:24.259127 17608 raft_consensus.cc:2229] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:24.259197 17608 raft_consensus.cc:2258] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:24.259266 17077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 10:26:24.259337 17609 ts_tablet_manager.cc:1905] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 10:26:24.259392 17077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:24.259802 17608 ts_tablet_manager.cc:1905] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:24.260474 17214 tablet_service.cc:1467] Processing CreateTablet for tablet 7819db2d80414bbf841c84f22efe77b8 (DEFAULT_TABLE table=test-table [id=1f31f87de9194894a2e3a96d3aef1954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:24.260578 17214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7819db2d80414bbf841c84f22efe77b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:24.260799 17609 ts_tablet_manager.cc:1918] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:24.260843 17609 log.cc:1196] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/1d459d3bac6c4a048d7a594d444579a7
I20260619 10:26:24.261014 17609 ts_tablet_manager.cc:1939] T 1d459d3bac6c4a048d7a594d444579a7 P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting consensus metadata
I20260619 10:26:24.261170 17608 ts_tablet_manager.cc:1918] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:24.261220 17608 log.cc:1196] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/1d459d3bac6c4a048d7a594d444579a7
I20260619 10:26:24.261410 17608 ts_tablet_manager.cc:1939] T 1d459d3bac6c4a048d7a594d444579a7 P 332de27cfcd34451a1675a68758bda05: Deleting consensus metadata
I20260619 10:26:24.261915 17064 catalog_manager.cc:4841] TS 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217): tablet 1d459d3bac6c4a048d7a594d444579a7 (table test-table [id=d634bc67ccf1473fa69ff3b2e781fb82]) successfully deleted
I20260619 10:26:24.261974 17064 catalog_manager.cc:4841] TS 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659): tablet 1d459d3bac6c4a048d7a594d444579a7 (table test-table [id=d634bc67ccf1473fa69ff3b2e781fb82]) successfully deleted
I20260619 10:26:24.262004 17590 tablet_bootstrap.cc:492] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c: Bootstrap starting.
I20260619 10:26:24.262069 17607 ts_tablet_manager.cc:1905] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:24.262089 17493 tablet_service.cc:1467] Processing CreateTablet for tablet 7819db2d80414bbf841c84f22efe77b8 (DEFAULT_TABLE table=test-table [id=1f31f87de9194894a2e3a96d3aef1954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:24.262174 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7819db2d80414bbf841c84f22efe77b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:24.262285 17590 tablet_bootstrap.cc:654] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:24.262540 17354 tablet_service.cc:1467] Processing CreateTablet for tablet 7819db2d80414bbf841c84f22efe77b8 (DEFAULT_TABLE table=test-table [id=1f31f87de9194894a2e3a96d3aef1954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:24.262624 17354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7819db2d80414bbf841c84f22efe77b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:24.263211 17586 tablet_bootstrap.cc:492] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: Bootstrap starting.
I20260619 10:26:24.263294 17607 ts_tablet_manager.cc:1918] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:24.263339 17607 log.cc:1196] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/1d459d3bac6c4a048d7a594d444579a7
I20260619 10:26:24.263523 17607 ts_tablet_manager.cc:1939] T 1d459d3bac6c4a048d7a594d444579a7 P f19e52fbd5ee4117aeb7e18f41f2018c: Deleting consensus metadata
I20260619 10:26:24.263566 17586 tablet_bootstrap.cc:654] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:24.263841 17585 tablet_bootstrap.cc:492] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: Bootstrap starting.
I20260619 10:26:24.263861 17062 catalog_manager.cc:4841] TS f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65:36801): tablet 1d459d3bac6c4a048d7a594d444579a7 (table test-table [id=d634bc67ccf1473fa69ff3b2e781fb82]) successfully deleted
I20260619 10:26:24.264118 17585 tablet_bootstrap.cc:654] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:24.264539 17586 tablet_bootstrap.cc:492] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: No bootstrap required, opened a new log
I20260619 10:26:24.264585 17586 ts_tablet_manager.cc:1397] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:24.264704 17590 tablet_bootstrap.cc:492] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c: No bootstrap required, opened a new log
I20260619 10:26:24.264701 17586 raft_consensus.cc:348] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.264746 17586 raft_consensus.cc:374] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:24.264760 17586 raft_consensus.cc:729] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332de27cfcd34451a1675a68758bda05, State: Initialized, Role: FOLLOWER
I20260619 10:26:24.264753 17590 ts_tablet_manager.cc:1397] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260619 10:26:24.264794 17586 consensus_queue.cc:260] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.264847 17590 raft_consensus.cc:348] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.264882 17586 ts_tablet_manager.cc:1428] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:24.264915 17590 raft_consensus.cc:374] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:24.264936 17590 raft_consensus.cc:729] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19e52fbd5ee4117aeb7e18f41f2018c, State: Initialized, Role: FOLLOWER
I20260619 10:26:24.264973 17590 consensus_queue.cc:260] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.265059 17590 ts_tablet_manager.cc:1428] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:24.265198 17585 tablet_bootstrap.cc:492] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: No bootstrap required, opened a new log
I20260619 10:26:24.265242 17585 ts_tablet_manager.cc:1397] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 10:26:24.265352 17585 raft_consensus.cc:348] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.265420 17585 raft_consensus.cc:374] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:24.265437 17585 raft_consensus.cc:729] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e504c6ffbe34d30a5bd82392b652b0e, State: Initialized, Role: FOLLOWER
I20260619 10:26:24.265472 17585 consensus_queue.cc:260] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.265558 17585 ts_tablet_manager.cc:1428] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:24.464636 17593 raft_consensus.cc:482] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:24.464744 17593 raft_consensus.cc:504] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.464910 17593 leader_election.cc:288] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659), 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217)
I20260619 10:26:24.465185 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7819db2d80414bbf841c84f22efe77b8" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332de27cfcd34451a1675a68758bda05" is_pre_election: true
I20260619 10:26:24.465164 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7819db2d80414bbf841c84f22efe77b8" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" is_pre_election: true
I20260619 10:26:24.465283 17374 raft_consensus.cc:2454] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 0.
I20260619 10:26:24.465283 17514 raft_consensus.cc:2454] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 0.
I20260619 10:26:24.465446 17168 leader_election.cc:302] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e504c6ffbe34d30a5bd82392b652b0e, f19e52fbd5ee4117aeb7e18f41f2018c; no voters:
I20260619 10:26:24.465545 17593 raft_consensus.cc:2781] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:24.465584 17593 raft_consensus.cc:482] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:24.465608 17593 raft_consensus.cc:3037] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:24.466328 17593 raft_consensus.cc:504] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.466434 17593 leader_election.cc:288] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 election: Requested vote from peers 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659), 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217)
I20260619 10:26:24.466609 17374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7819db2d80414bbf841c84f22efe77b8" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e504c6ffbe34d30a5bd82392b652b0e"
I20260619 10:26:24.466636 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7819db2d80414bbf841c84f22efe77b8" candidate_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332de27cfcd34451a1675a68758bda05"
I20260619 10:26:24.466691 17374 raft_consensus.cc:3037] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:24.466696 17514 raft_consensus.cc:3037] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:24.467407 17514 raft_consensus.cc:2454] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 1.
I20260619 10:26:24.467407 17374 raft_consensus.cc:2454] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19e52fbd5ee4117aeb7e18f41f2018c in term 1.
I20260619 10:26:24.467573 17168 leader_election.cc:302] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 332de27cfcd34451a1675a68758bda05, f19e52fbd5ee4117aeb7e18f41f2018c; no voters:
I20260619 10:26:24.467676 17593 raft_consensus.cc:2781] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:24.467753 17593 raft_consensus.cc:686] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 LEADER]: Becoming Leader. State: Replica: f19e52fbd5ee4117aeb7e18f41f2018c, State: Running, Role: LEADER
I20260619 10:26:24.467803 17593 consensus_queue.cc:237] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } }
I20260619 10:26:24.468394 17075 catalog_manager.cc:5495] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c reported cstate change: term changed from 0 to 1, leader changed from <none> to f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65). New cstate: current_term: 1 leader_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19e52fbd5ee4117aeb7e18f41f2018c" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 } health_report { overall_health: UNKNOWN } } }
I20260619 10:26:24.496280 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.494235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.9.90.65:36801)
0619 10:26:24.494279 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.494281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.494312 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:26:24.494332 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.494333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.494342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.494378 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:24.494476 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.494957 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.495756 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.495945 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.495996 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.496097 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.496124 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.496153 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.496202 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.496217 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:26:24.499946 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.498117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:51153 (local address 127.9.90.65:36801)
0619 10:26:24.498147 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:24.498149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:24.498154 (+ 5us) server_negotiation.cc:407] Connection header received
0619 10:26:24.498170 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:24.498171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:24.498177 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:24.498230 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:24.498312 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.498760 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.499436 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.499619 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.499665 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.499767 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.499789 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.499820 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:24.499859 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 10:26:24.499878 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":15}
I20260619 10:26:24.499953 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.498024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36801 (local address 127.9.90.65:51153)
0619 10:26:24.498076 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:24.498086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:24.498123 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:24.498241 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:24.498242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:24.498243 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 10:26:24.498277 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.498278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.498815 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.498816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.499378 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:24.499379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:24.499678 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:24.499678 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:24.499713 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:24.499718 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 10:26:24.499829 (+ 111us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 10:26:24.499830 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 10:26:24.499858 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 10:26:24.499891 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:26:24.501653 17514 raft_consensus.cc:1261] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Refusing update from remote peer f19e52fbd5ee4117aeb7e18f41f2018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:24.501816 17374 raft_consensus.cc:1261] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Refusing update from remote peer f19e52fbd5ee4117aeb7e18f41f2018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:24.501884 17634 consensus_queue.cc:1035] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "332de27cfcd34451a1675a68758bda05" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 43217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:24.501997 17595 consensus_queue.cc:1035] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e504c6ffbe34d30a5bd82392b652b0e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:24.511030 17075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:24.511122 17075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:24.511622 17075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f4cda0a95a14915b68059ca4f5b5c88: Sending DeleteTablet for 3 replicas of tablet 7819db2d80414bbf841c84f22efe77b8
I20260619 10:26:24.511816 17214 tablet_service.cc:1514] Processing DeleteTablet for tablet 7819db2d80414bbf841c84f22efe77b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:24 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:38930
I20260619 10:26:24.511831 17354 tablet_service.cc:1514] Processing DeleteTablet for tablet 7819db2d80414bbf841c84f22efe77b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:24 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:42172
I20260619 10:26:24.511852 17493 tablet_service.cc:1514] Processing DeleteTablet for tablet 7819db2d80414bbf841c84f22efe77b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:24 UTC) from {username='oryx', principal='oryx/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46706
I20260619 10:26:24.511912 17609 tablet_replica.cc:330] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: stopping tablet replica
I20260619 10:26:24.511912 17607 tablet_replica.cc:330] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c: stopping tablet replica
I20260619 10:26:24.511955 17609 raft_consensus.cc:2229] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:24.511962 17607 raft_consensus.cc:2229] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:24.511984 17609 raft_consensus.cc:2258] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:24.511998 17608 tablet_replica.cc:330] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: stopping tablet replica
I20260619 10:26:24.512035 17607 raft_consensus.cc:2258] T 7819db2d80414bbf841c84f22efe77b8 P f19e52fbd5ee4117aeb7e18f41f2018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:24.512043 17608 raft_consensus.cc:2229] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:24.512072 17608 raft_consensus.cc:2258] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:24.512142 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 17149
I20260619 10:26:24.512295 17609 ts_tablet_manager.cc:1905] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:24.513496 17608 ts_tablet_manager.cc:1905] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:24.513705 17609 ts_tablet_manager.cc:1918] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:24.513752 17609 log.cc:1196] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/7819db2d80414bbf841c84f22efe77b8
I20260619 10:26:24.513962 17609 ts_tablet_manager.cc:1939] T 7819db2d80414bbf841c84f22efe77b8 P 6e504c6ffbe34d30a5bd82392b652b0e: Deleting consensus metadata
I20260619 10:26:24.514290 17064 catalog_manager.cc:4841] TS 6e504c6ffbe34d30a5bd82392b652b0e (127.9.90.66:39659): tablet 7819db2d80414bbf841c84f22efe77b8 (table test-table [id=1f31f87de9194894a2e3a96d3aef1954]) successfully deleted
I20260619 10:26:24.514598 17608 ts_tablet_manager.cc:1918] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 10:26:24.514632 17608 log.cc:1196] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/7819db2d80414bbf841c84f22efe77b8
I20260619 10:26:24.514751 17608 ts_tablet_manager.cc:1939] T 7819db2d80414bbf841c84f22efe77b8 P 332de27cfcd34451a1675a68758bda05: Deleting consensus metadata
I20260619 10:26:24.515089 17064 catalog_manager.cc:4841] TS 332de27cfcd34451a1675a68758bda05 (127.9.90.67:43217): tablet 7819db2d80414bbf841c84f22efe77b8 (table test-table [id=1f31f87de9194894a2e3a96d3aef1954]) successfully deleted
W20260619 10:26:24.517975 17062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:36801 (error 108)
I20260619 10:26:24.518232 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 17289
I20260619 10:26:24.521401 17674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:24.518291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36801 (local address 127.0.0.1:38980)
0619 10:26:24.521323 (+ 3032us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:24.521369 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:36801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3017,"thread_start_us":34,"threads_started":1}
W20260619 10:26:24.521552 17062 catalog_manager.cc:4584] TS f19e52fbd5ee4117aeb7e18f41f2018c (127.9.90.65:36801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7819db2d80414bbf841c84f22efe77b8: Network error: Client connection negotiation failed: client connection to 127.9.90.65:36801: connect: Connection refused (error 111)
I20260619 10:26:24.523164 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 17429
I20260619 10:26:24.528504 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 17045
2026-06-19T10:26:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2567 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:24 dist-test-slave-hw7q krb5kdc[17677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:24 dist-test-slave-hw7q krb5kdc[17677](info): set up 2 sockets
Jun 19 10:26:24 dist-test-slave-hw7q krb5kdc[17677](info): commencing operation
krb5kdc: starting...
W20260619 10:26:25.490099 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:25 dist-test-slave-hw7q krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:25.594585 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33413
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:33413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:25.671121 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:25.671321 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:25.671342 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:25.672497 17690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:25.672540 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:25.672560 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:25.672583 17690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:25.672602 17690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:25.674057 17690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:33413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:25.674254 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:25.674422 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:25.678972 17698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:25.679013 17695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:25.679023 17696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:25.679299 17690 server_base.cc:1034] running on GCE node
I20260619 10:26:25.679538 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:25.679790 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:25.680917 17690 hybrid_clock.cc:648] HybridClock initialized: now 1781864785680909 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T10:26:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1153 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:25 dist-test-slave-hw7q krb5kdc[17703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:25 dist-test-slave-hw7q krb5kdc[17703](info): set up 2 sockets
Jun 19 10:26:25 dist-test-slave-hw7q krb5kdc[17703](info): commencing operation
krb5kdc: starting...
W20260619 10:26:26.630311 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:26 dist-test-slave-hw7q krb5kdc[17703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:26.744732 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40493
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:40493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:26.824152 17716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:26.824353 17716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:26.824374 17716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:26.825500 17716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:26.825541 17716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:26.825553 17716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:26.825567 17716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:26.825590 17716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:26.827030 17716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:40493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:26.827240 17716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:26.827435 17716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:26.832010 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:26.832137 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:26.832311 17716 server_base.cc:1034] running on GCE node
W20260619 10:26:26.832358 17722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:26.832597 17716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:26.832908 17716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:26.834050 17716 hybrid_clock.cc:648] HybridClock initialized: now 1781864786834025 us; error 37 us; skew 500 ppm
Jun 19 10:26:26 dist-test-slave-hw7q krb5kdc[17703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864786, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:26.836457 17716 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:26.836862 17716 webserver.cc:460] Webserver started at http://127.9.90.126:42559/ using document root <none> and password file <none>
I20260619 10:26:26.837055 17716 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:26.837100 17716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:26.837219 17716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:26.838336 17716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "3b3cb79f5f054a758c854c4a9bc5be1c"
format_stamp: "Formatted at 2026-06-19 10:26:26 on dist-test-slave-hw7q"
server_key: "82db3681594fbe02c6fac1683d11f4d7"
server_key_iv: "1688e3122874ddafe978947d85bcd170"
server_key_version: "encryptionkey@0"
I20260619 10:26:26.838680 17716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "3b3cb79f5f054a758c854c4a9bc5be1c"
format_stamp: "Formatted at 2026-06-19 10:26:26 on dist-test-slave-hw7q"
server_key: "82db3681594fbe02c6fac1683d11f4d7"
server_key_iv: "1688e3122874ddafe978947d85bcd170"
server_key_version: "encryptionkey@0"
I20260619 10:26:26.839882 17716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:26.840473 17731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:26.840629 17716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:26.840708 17716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "3b3cb79f5f054a758c854c4a9bc5be1c"
format_stamp: "Formatted at 2026-06-19 10:26:26 on dist-test-slave-hw7q"
server_key: "82db3681594fbe02c6fac1683d11f4d7"
server_key_iv: "1688e3122874ddafe978947d85bcd170"
server_key_version: "encryptionkey@0"
I20260619 10:26:26.840770 17716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:26.850108 17716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:26.851769 17716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:26.851888 17716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:26.855679 17716 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:40493
I20260619 10:26:26.855726 17783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:40493 every 8 connection(s)
I20260619 10:26:26.856014 17716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:26.856772 17784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:26.858963 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 17716
I20260619 10:26:26.858994 17784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: Bootstrap starting.
I20260619 10:26:26.859053 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:26.859256 9577 external_mini_cluster.cc:1428] Setting key a8f11cab73659428ecd0eb42173bdefd
I20260619 10:26:26.859550 17784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:26.859870 17784 log.cc:824] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:26.860493 17784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: No bootstrap required, opened a new log
I20260619 10:26:26.862645 17784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } }
I20260619 10:26:26.862804 17784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:26.862834 17784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3cb79f5f054a758c854c4a9bc5be1c, State: Initialized, Role: FOLLOWER
I20260619 10:26:26.862944 17784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } }
I20260619 10:26:26.863006 17784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:26.863040 17784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:26.863082 17784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:26.863595 17784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } }
I20260619 10:26:26.863698 17784 leader_election.cc:302] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b3cb79f5f054a758c854c4a9bc5be1c; no voters:
I20260619 10:26:26.863852 17784 leader_election.cc:288] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:26.863898 17789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:26.864063 17789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 1 LEADER]: Becoming Leader. State: Replica: 3b3cb79f5f054a758c854c4a9bc5be1c, State: Running, Role: LEADER
I20260619 10:26:26.864087 17784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:26.864176 17789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } }
I20260619 10:26:26.864466 17789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b3cb79f5f054a758c854c4a9bc5be1c. Latest consensus state: current_term: 1 leader_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } }
I20260619 10:26:26.864475 17790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } }
I20260619 10:26:26.864568 17789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:26.864573 17790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:26.864876 17795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:26.865521 17795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 10:26:26 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:26.867475 17795 catalog_manager.cc:1347] Generated new cluster ID: 0d610a525b3f4176a7e333cec04c8ddd
I20260619 10:26:26.867530 17795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:26.869067 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:26.859697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.9.90.126:40493)
0619 10:26:26.859846 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:26.859854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:26.859893 (+ 39us) server_negotiation.cc:407] Connection header received
0619 10:26:26.860303 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:26.860310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:26.860356 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:26.860463 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:26.861178 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:26.862126 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:26.863145 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:26.863547 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:26.863631 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:26.866294 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:26.866299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:26.866299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:26.866310 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:26.867759 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:26.868052 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:26.868055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:26.868055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:26.868134 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:26.868301 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:26.868302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:26.868302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:26.868519 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:26.868637 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:26.868751 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 10:26:26.868876 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260619 10:26:26.876066 17795 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:26.876768 17795 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:26.888218 17795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: Generated new TSK 0
I20260619 10:26:26.888440 17795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:26:26.915695 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40493
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:26.993988 17810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:26.994184 17810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:26.994215 17810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:26.995359 17810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:26.995399 17810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:26.995432 17810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:26.996831 17810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:26.997049 17810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:26.997254 17810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:27.002005 17816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.002002 17818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.002070 17815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:27.002576 17810 server_base.cc:1034] running on GCE node
I20260619 10:26:27.002784 17810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:27.002998 17810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:27.004130 17810 hybrid_clock.cc:648] HybridClock initialized: now 1781864787004114 us; error 31 us; skew 500 ppm
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:27.007117 17810 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:26:27.007668 17810 webserver.cc:460] Webserver started at http://127.9.90.65:36741/ using document root <none> and password file <none>
I20260619 10:26:27.007936 17810 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:27.007997 17810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:27.008121 17810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:27.009334 17810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "a18b4395091848f6b8f5cd9b9e096d4f"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "0d7b88d091e1c63eb8d5ab9a341ffb2f"
server_key_iv: "cae54e15afb810aaed8b4e586374f017"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.009729 17810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "a18b4395091848f6b8f5cd9b9e096d4f"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "0d7b88d091e1c63eb8d5ab9a341ffb2f"
server_key_iv: "cae54e15afb810aaed8b4e586374f017"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.011271 17810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:27.011946 17825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.012086 17810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:27.012162 17810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "a18b4395091848f6b8f5cd9b9e096d4f"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "0d7b88d091e1c63eb8d5ab9a341ffb2f"
server_key_iv: "cae54e15afb810aaed8b4e586374f017"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.012213 17810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:27.040292 17810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:27.042213 17810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:27.042349 17810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:27.042546 17810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:27.042843 17810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:27.042876 17810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.042899 17810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:27.042923 17810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.048425 17810 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:34109
I20260619 10:26:27.048482 17938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:34109 every 8 connection(s)
I20260619 10:26:27.048780 17810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:27.050549 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 17810
I20260619 10:26:27.050635 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:27.050808 9577 external_mini_cluster.cc:1428] Setting key 2751a2fabbcbec1492ff81b01e35d105
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:27.057941 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.049466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:41139 (local address 127.9.90.126:40493)
0619 10:26:27.049546 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.049549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.049680 (+ 131us) server_negotiation.cc:407] Connection header received
0619 10:26:27.050162 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.050163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.050173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.050212 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.051214 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.051701 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.052692 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.052899 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.052955 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.055315 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:27.055317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:27.055317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:27.055324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:27.056556 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:27.057011 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:27.057012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:27.057012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:27.057061 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:27.057306 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:27.057307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:27.057307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:27.057403 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:27.057450 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:27.057775 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 10:26:27.057874 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 10:26:27.058187 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.049425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:41139)
0619 10:26:27.049559 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.049598 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.050073 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.050284 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.050287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.050907 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.051118 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.051123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.051800 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.051801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.052606 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.052608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.052968 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.052969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.053092 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.053608 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.053617 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.055225 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:27.056635 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:27.056641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:27.056644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:27.056939 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:27.057112 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:27.057113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:27.057114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:27.057246 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:27.057460 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:27.057465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:27.057632 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 10:26:27.057766 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 10:26:27.057960 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
I20260619 10:26:27.058704 17939 heartbeater.cc:347] Connected to a master server at 127.9.90.126:40493
I20260619 10:26:27.058789 17939 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:27.058954 17939 heartbeater.cc:511] Master 127.9.90.126:40493 requested a full tablet report, sending...
I20260619 10:26:27.059454 17748 ts_manager.cc:194] Registered new tserver with Master: a18b4395091848f6b8f5cd9b9e096d4f (127.9.90.65:34109)
I20260619 10:26:27.060174 17748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:41139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:26:27.073541 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40493
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:27.150731 17944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:27.150933 17944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:27.150955 17944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:27.152073 17944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:27.152107 17944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:27.152130 17944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:27.153496 17944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:27.153676 17944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:27.153878 17944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:27.158538 17949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.158644 17952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.158736 17950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:27.158789 17944 server_base.cc:1034] running on GCE node
I20260619 10:26:27.158991 17944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:27.159205 17944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:27.160367 17944 hybrid_clock.cc:648] HybridClock initialized: now 1781864787160354 us; error 33 us; skew 500 ppm
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:27.162703 17944 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:26:27.163133 17944 webserver.cc:460] Webserver started at http://127.9.90.66:33325/ using document root <none> and password file <none>
I20260619 10:26:27.163349 17944 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:27.163396 17944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:27.163516 17944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:27.164554 17944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "457305ffaa5947aaae495a323136748f"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "9a93987d6254f0c8d3e62cc986a06466"
server_key_iv: "cd02189eed7efa7e0cb5b45605acd205"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.164868 17944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "457305ffaa5947aaae495a323136748f"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "9a93987d6254f0c8d3e62cc986a06466"
server_key_iv: "cd02189eed7efa7e0cb5b45605acd205"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.166236 17944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:27.166916 17959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.167068 17944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:27.167157 17944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "457305ffaa5947aaae495a323136748f"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "9a93987d6254f0c8d3e62cc986a06466"
server_key_iv: "cd02189eed7efa7e0cb5b45605acd205"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.167232 17944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:27.187757 17944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:27.189687 17944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:27.189832 17944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:27.190052 17944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:27.190392 17944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:27.190426 17944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.190447 17944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:27.190471 17944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.195988 17944 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:41083
I20260619 10:26:27.196038 18072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:41083 every 8 connection(s)
I20260619 10:26:27.196341 17944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:27.197623 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 17944
I20260619 10:26:27.197721 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:27.197908 9577 external_mini_cluster.cc:1428] Setting key b0b9b257487edae2f9cc06e3ac8a4e4c
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:27.205498 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.196990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:48331 (local address 127.9.90.126:40493)
0619 10:26:27.197048 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.197051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.197242 (+ 191us) server_negotiation.cc:407] Connection header received
0619 10:26:27.197813 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.197815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.197827 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.197876 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.198992 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.199473 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.200421 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.200706 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.200763 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.202834 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:27.202836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:27.202836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:27.202844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:27.204250 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:27.204612 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:27.204613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:27.204614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:27.204677 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:27.204881 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:27.204882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:27.204882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:27.205020 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:27.205093 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:27.205327 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 10:26:27.205406 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
I20260619 10:26:27.205575 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.196961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:48331)
0619 10:26:27.197106 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.197145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.197681 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.197949 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.197955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.198713 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.198906 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.198911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.199552 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.199553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.200344 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.200346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.200785 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.200786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.200900 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.201361 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.201370 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.202744 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:27.204330 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:27.204336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:27.204337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:27.204544 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:27.204739 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:27.204739 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:27.204739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:27.204827 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:27.205079 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:27.205082 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:27.205194 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 10:26:27.205289 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 10:26:27.205406 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260619 10:26:27.206226 18073 heartbeater.cc:347] Connected to a master server at 127.9.90.126:40493
I20260619 10:26:27.206312 18073 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:27.206485 18073 heartbeater.cc:511] Master 127.9.90.126:40493 requested a full tablet report, sending...
I20260619 10:26:27.206856 17748 ts_manager.cc:194] Registered new tserver with Master: 457305ffaa5947aaae495a323136748f (127.9.90.66:41083)
I20260619 10:26:27.207268 17748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:48331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:26:27.222009 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40493
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:27.299674 18078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:27.299861 18078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:27.299890 18078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:27.301061 18078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:27.301100 18078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:27.301136 18078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:27.302556 18078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:27.302783 18078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:27.302982 18078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:27.307564 18083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.307559 18086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.307592 18084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:27.307898 18078 server_base.cc:1034] running on GCE node
I20260619 10:26:27.308094 18078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:27.308313 18078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:27.309445 18078 hybrid_clock.cc:648] HybridClock initialized: now 1781864787309425 us; error 36 us; skew 500 ppm
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:27.311754 18078 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:26:27.312168 18078 webserver.cc:460] Webserver started at http://127.9.90.67:34261/ using document root <none> and password file <none>
I20260619 10:26:27.312379 18078 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:27.312425 18078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:27.312546 18078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:27.313544 18078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "0f2de157b5864e6c97b908880aff08ec"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "92af308e988bf8dfc8be3a31ad5f762f"
server_key_iv: "438f3feff1685bdceaed7efcdec178e3"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.313903 18078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "0f2de157b5864e6c97b908880aff08ec"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "92af308e988bf8dfc8be3a31ad5f762f"
server_key_iv: "438f3feff1685bdceaed7efcdec178e3"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.315120 18078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:27.315822 18093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.315980 18078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:27.316080 18078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "0f2de157b5864e6c97b908880aff08ec"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "92af308e988bf8dfc8be3a31ad5f762f"
server_key_iv: "438f3feff1685bdceaed7efcdec178e3"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.316143 18078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:27.342319 18078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:27.344235 18078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:27.344368 18078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:27.344574 18078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:27.344867 18078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:27.344900 18078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.344929 18078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:27.344952 18078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.350562 18078 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:43445
I20260619 10:26:27.350615 18206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:43445 every 8 connection(s)
I20260619 10:26:27.350926 18078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:26:27.356351 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 18078
I20260619 10:26:27.356438 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:27.356606 9577 external_mini_cluster.cc:1428] Setting key b8851aa4b2a1d2f5e294101b87755c05
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:27.359742 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.351633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:51407 (local address 127.9.90.126:40493)
0619 10:26:27.351684 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.351686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.351908 (+ 222us) server_negotiation.cc:407] Connection header received
0619 10:26:27.352401 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.352402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.352412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.352450 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.353337 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.353914 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.354879 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.355068 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.355129 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.357166 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:27.357168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:27.357168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:27.357175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:27.358447 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:27.358881 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:27.358882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:27.358882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:27.358941 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:27.359198 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:27.359199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:27.359199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:27.359319 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:27.359382 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:27.359594 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 10:26:27.359672 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:27.359941 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.351618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:51407)
0619 10:26:27.351784 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.351821 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.352309 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.352512 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.352515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.353066 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.353264 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.353269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.353982 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.353983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.354785 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.354787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.355143 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.355144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.355281 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.355731 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.355739 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.357086 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:27.358548 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:27.358554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:27.358556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:27.358794 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:27.359006 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:27.359007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:27.359007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:27.359108 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:27.359376 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:27.359379 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:27.359495 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 10:26:27.359572 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 10:26:27.359693 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260619 10:26:27.360349 18207 heartbeater.cc:347] Connected to a master server at 127.9.90.126:40493
I20260619 10:26:27.360442 18207 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:27.360616 18207 heartbeater.cc:511] Master 127.9.90.126:40493 requested a full tablet report, sending...
I20260619 10:26:27.360963 17748 ts_manager.cc:194] Registered new tserver with Master: 0f2de157b5864e6c97b908880aff08ec (127.9.90.67:43445)
I20260619 10:26:27.361333 17748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:51407
I20260619 10:26:27.368603 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab.
Entry for principal oryx/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab.
WARNING: no policy specified for oryx/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab.
Entry for principal oryx/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab.
WARNING: no policy specified for oryx/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab.
Entry for principal oryx/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab.
I20260619 10:26:27.432307 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 17810
I20260619 10:26:27.436092 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 17944
I20260619 10:26:27.439647 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 18078
I20260619 10:26:27.443041 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 17716
I20260619 10:26:27.447106 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40493
--webserver_interface=127.9.90.126
--webserver_port=42559
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:40493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:27.525234 18216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:27.525442 18216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:27.525465 18216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:27.526639 18216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:27.526686 18216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:27.526700 18216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:27.526711 18216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:27.526721 18216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:27.528144 18216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:40493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=42559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:27.528342 18216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:27.528515 18216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:27.532987 18221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.532997 18222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.532984 18224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:27.533272 18216 server_base.cc:1034] running on GCE node
I20260619 10:26:27.533483 18216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:27.533725 18216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:27.534859 18216 hybrid_clock.cc:648] HybridClock initialized: now 1781864787534849 us; error 26 us; skew 500 ppm
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:27.537109 18216 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:27.537489 18216 webserver.cc:460] Webserver started at http://127.9.90.126:42559/ using document root <none> and password file <none>
I20260619 10:26:27.537657 18216 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:27.537704 18216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:27.539160 18216 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:27.539738 18231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.539898 18216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:27.539975 18216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "3b3cb79f5f054a758c854c4a9bc5be1c"
format_stamp: "Formatted at 2026-06-19 10:26:26 on dist-test-slave-hw7q"
server_key: "82db3681594fbe02c6fac1683d11f4d7"
server_key_iv: "1688e3122874ddafe978947d85bcd170"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.540310 18216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:27.554171 18216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:27.555814 18216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:27.555919 18216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:27.559489 18216 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:40493
I20260619 10:26:27.559537 18283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:40493 every 8 connection(s)
I20260619 10:26:27.559863 18216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:27.561203 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 18216
I20260619 10:26:27.561239 18284 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:26:27.561457 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:34109
--local_ip_for_outbound_sockets=127.9.90.65
--tserver_master_addrs=127.9.90.126:40493
--webserver_port=36741
--webserver_interface=127.9.90.65
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.90.65
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:26:27.561928 18284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: Bootstrap starting.
I20260619 10:26:27.569655 18284 log.cc:824] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:27.571265 18284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:26:27.571506 18284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: Bootstrap complete.
I20260619 10:26:27.573480 18284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } }
I20260619 10:26:27.573704 18284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3cb79f5f054a758c854c4a9bc5be1c, State: Initialized, Role: FOLLOWER
I20260619 10:26:27.573880 18284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } }
I20260619 10:26:27.573943 18284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:27.573989 18284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:27.574043 18284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:26:27.575029 18284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } }
I20260619 10:26:27.575130 18284 leader_election.cc:302] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b3cb79f5f054a758c854c4a9bc5be1c; no voters:
I20260619 10:26:27.575286 18284 leader_election.cc:288] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 10:26:27.575330 18288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:26:27.575448 18288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [term 2 LEADER]: Becoming Leader. State: Replica: 3b3cb79f5f054a758c854c4a9bc5be1c, State: Running, Role: LEADER
I20260619 10:26:27.575542 18284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:27.575541 18288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } }
I20260619 10:26:27.575850 18288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b3cb79f5f054a758c854c4a9bc5be1c. Latest consensus state: current_term: 2 leader_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } }
I20260619 10:26:27.575917 18288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:27.576251 18289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3cb79f5f054a758c854c4a9bc5be1c" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } }
I20260619 10:26:27.576345 18289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:27.576383 18293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:27.576858 18293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:27.577001 18293 catalog_manager.cc:1259] Loaded cluster ID: 0d610a525b3f4176a7e333cec04c8ddd
I20260619 10:26:27.577052 18293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:27.578032 18293 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:27.578187 18293 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b3cb79f5f054a758c854c4a9bc5be1c: Loaded TSK: 0
I20260619 10:26:27.578374 18293 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:26:27.654621 18286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:27.654832 18286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:27.654865 18286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:27.656111 18286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:27.656152 18286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:27.656188 18286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:27.657601 18286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.65.keytab
--principal=oryx/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:34109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=36741
--tserver_master_addrs=127.9.90.126:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:27.657850 18286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:27.658056 18286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:27.663689 18311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.663647 18310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.663789 18313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:27.663929 18286 server_base.cc:1034] running on GCE node
I20260619 10:26:27.664260 18286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:27.664489 18286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:27.665644 18286 hybrid_clock.cc:648] HybridClock initialized: now 1781864787665623 us; error 39 us; skew 500 ppm
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:27.668210 18286 init.cc:377] Logged in from keytab as oryx/127.9.90.65@KRBTEST.COM (short username oryx)
I20260619 10:26:27.668653 18286 webserver.cc:460] Webserver started at http://127.9.90.65:36741/ using document root <none> and password file <none>
I20260619 10:26:27.668880 18286 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:27.668941 18286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:27.670327 18286 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 10:26:27.670956 18320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.671110 18286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:26:27.671193 18286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "a18b4395091848f6b8f5cd9b9e096d4f"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "0d7b88d091e1c63eb8d5ab9a341ffb2f"
server_key_iv: "cae54e15afb810aaed8b4e586374f017"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.671517 18286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:27.680090 18286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:27.682267 18286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:27.682412 18286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:27.682646 18286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:27.682928 18286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:27.682963 18286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.682992 18286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:27.683015 18286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.688905 18286 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:34109
I20260619 10:26:27.688961 18433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:34109 every 8 connection(s)
I20260619 10:26:27.689287 18286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:27.696406 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 18286
I20260619 10:26:27.696717 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:41083
--local_ip_for_outbound_sockets=127.9.90.66
--tserver_master_addrs=127.9.90.126:40493
--webserver_port=33325
--webserver_interface=127.9.90.66
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.90.66
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:27.701915 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.690005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:45863)
0619 10:26:27.690272 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.690320 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.690777 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.693416 (+ 2639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.693419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.694049 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.694265 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.694271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.695929 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.695930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.697111 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.697115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.698039 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.698040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.698199 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.698828 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.698838 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.701622 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
W20260619 10:26:27.702212 18434 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:40493 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.90.126:40493: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
W20260619 10:26:27.702355 18437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:27.690094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:45863 (local address 127.9.90.126:40493)
0619 10:26:27.692940 (+ 2846us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.692950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.693000 (+ 50us) server_negotiation.cc:407] Connection header received
0619 10:26:27.693131 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.693139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.693197 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.693349 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.694396 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.695804 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.697215 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.697892 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.698039 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.701842 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:45863: BlockingRecv error: recv got EOF from 127.9.90.65:45863 (error 108)
Metrics: {"server-negotiator.queue_time_us":2784,"thread_start_us":86,"threads_started":1}
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:27.708063 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.702436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:52787)
0619 10:26:27.702528 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.702538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.702574 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.703017 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.703019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.703165 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.703220 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.703222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.704158 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.704159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.704977 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.704980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.705458 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.705459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.705524 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.706020 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.706025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.707980 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 10:26:27.708163 18439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:27.702697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:52787 (local address 127.9.90.126:40493)
0619 10:26:27.702875 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.702877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.702885 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:27.702910 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.702911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.702923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.702996 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.703300 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.704076 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.705061 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.705381 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.705455 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.708091 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:52787: BlockingRecv error: recv got EOF from 127.9.90.65:52787 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:27.713593 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.708555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:59125)
0619 10:26:27.708674 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.708693 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.708750 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.708880 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.708881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.709015 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.709047 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.709049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.709888 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.709889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.710723 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.710732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.711177 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.711178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.711229 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.711667 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.711672 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.713522 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20260619 10:26:27.713671 18439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:27.708633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:59125 (local address 127.9.90.126:40493)
0619 10:26:27.708687 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.708689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.708737 (+ 48us) server_negotiation.cc:407] Connection header received
0619 10:26:27.708795 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.708796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.708812 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.708870 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.709112 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.709823 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.710805 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.711097 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.711170 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.713613 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:59125: BlockingRecv error: recv got EOF from 127.9.90.65:59125 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 10:26:27.713930 18434 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:26:27.779192 18438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:27.779412 18438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:27.779433 18438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:27.780614 18438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:27.780659 18438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:27.780690 18438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:27.782191 18438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.66.keytab
--principal=oryx/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:41083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=33325
--tserver_master_addrs=127.9.90.126:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:27.782413 18438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:27.782622 18438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:27.787230 18445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.787266 18447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.787253 18444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:27.787441 18438 server_base.cc:1034] running on GCE node
I20260619 10:26:27.787634 18438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:27.787854 18438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:27.788995 18438 hybrid_clock.cc:648] HybridClock initialized: now 1781864787788980 us; error 31 us; skew 500 ppm
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:27.791402 18438 init.cc:377] Logged in from keytab as oryx/127.9.90.66@KRBTEST.COM (short username oryx)
I20260619 10:26:27.791823 18438 webserver.cc:460] Webserver started at http://127.9.90.66:33325/ using document root <none> and password file <none>
I20260619 10:26:27.792008 18438 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:27.792059 18438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:27.793322 18438 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:27.793844 18454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.793984 18438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.794060 18438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "457305ffaa5947aaae495a323136748f"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "9a93987d6254f0c8d3e62cc986a06466"
server_key_iv: "cd02189eed7efa7e0cb5b45605acd205"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.794399 18438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:27.803339 18438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:27.805064 18438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:27.805186 18438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:27.805400 18438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:27.805683 18438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:27.805714 18438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.805737 18438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:27.805760 18438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.811288 18438 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:41083
I20260619 10:26:27.811357 18567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:41083 every 8 connection(s)
I20260619 10:26:27.811663 18438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 10:26:27.818521 18439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:27.812321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:53589 (local address 127.9.90.126:40493)
0619 10:26:27.812376 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.812378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.812584 (+ 206us) server_negotiation.cc:407] Connection header received
0619 10:26:27.813088 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.813089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.813105 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.813156 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.814111 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.814611 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.815579 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.815792 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.815846 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.818465 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:53589: BlockingRecv error: recv got EOF from 127.9.90.66:53589 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:26:27.818569 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.812307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:53589)
0619 10:26:27.812465 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.812502 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.812984 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.813236 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.813240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.813812 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.814022 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.814027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.814694 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.814695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.815490 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.815492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.815862 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.815862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.815972 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.816393 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.816401 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.818371 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
W20260619 10:26:27.818761 18568 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:40493 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.90.126:40493: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
I20260619 10:26:27.821765 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 18438
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:27.822077 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:43445
--local_ip_for_outbound_sockets=127.9.90.67
--tserver_master_addrs=127.9.90.126:40493
--webserver_port=34261
--webserver_interface=127.9.90.67
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.90.67
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:27.822916 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.818954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:43919)
0619 10:26:27.819043 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.819053 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.819096 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.819190 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.819190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.819314 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.819365 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.819366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.819925 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.819926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.820487 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.820489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.820801 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.820801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.820844 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.821191 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.821195 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.822858 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 10:26:27.822979 18439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:27.819002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:43919 (local address 127.9.90.126:40493)
0619 10:26:27.819045 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.819046 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.819085 (+ 39us) server_negotiation.cc:407] Connection header received
0619 10:26:27.819133 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.819134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.819143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.819180 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.819424 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.819870 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.820561 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.820748 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.820795 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.822927 (+ 2132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:43919: BlockingRecv error: recv got EOF from 127.9.90.66:43919 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:27.827111 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.823325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:55943)
0619 10:26:27.823370 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.823382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.823421 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.823554 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.823554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.823646 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.823670 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.823671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.824438 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.824438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.824907 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.824908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.825194 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.825195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.825254 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.825609 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.825613 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.827053 (+ 1440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:26:27.827205 18439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:27.823376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:55943 (local address 127.9.90.126:40493)
0619 10:26:27.823409 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.823411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.823420 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:26:27.823490 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.823490 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.823499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.823539 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.823717 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.824374 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.824959 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.825133 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.825185 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.827157 (+ 1972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:55943: BlockingRecv error: recv got EOF from 127.9.90.66:55943 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:26:27.827325 18568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:26:27.905893 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:27.906100 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:27.906124 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:27.907320 18571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:27.907361 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:27.907387 18571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:27.908815 18571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/krb5kdc/oryx_127.9.90.67.keytab
--principal=oryx/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:43445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=34261
--tserver_master_addrs=127.9.90.126:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:27.909036 18571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:27.909262 18571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:26:27.914413 18571 server_base.cc:1034] running on GCE node
W20260619 10:26:27.914317 18579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.914332 18577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:27.914317 18576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:27.914834 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:27.915073 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:27.916216 18571 hybrid_clock.cc:648] HybridClock initialized: now 1781864787916186 us; error 45 us; skew 500 ppm
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864787, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:27.918679 18571 init.cc:377] Logged in from keytab as oryx/127.9.90.67@KRBTEST.COM (short username oryx)
I20260619 10:26:27.919090 18571 webserver.cc:460] Webserver started at http://127.9.90.67:34261/ using document root <none> and password file <none>
I20260619 10:26:27.919268 18571 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:27.919317 18571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:27.920611 18571 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:27.921151 18586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.921326 18571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:27.921406 18571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "0f2de157b5864e6c97b908880aff08ec"
format_stamp: "Formatted at 2026-06-19 10:26:27 on dist-test-slave-hw7q"
server_key: "92af308e988bf8dfc8be3a31ad5f762f"
server_key_iv: "438f3feff1685bdceaed7efcdec178e3"
server_key_version: "encryptionkey@0"
I20260619 10:26:27.921742 18571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:27.936509 18571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:27.938756 18571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:27.938872 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:27.939072 18571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:27.939388 18571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:27.939421 18571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.939451 18571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:27.939474 18571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:27.945142 18571 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:43445
I20260619 10:26:27.945226 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:43445 every 8 connection(s)
I20260619 10:26:27.945533 18571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:27.952930 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.946269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:36531)
0619 10:26:27.946414 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.946451 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.946930 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.947218 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.947222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.947780 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.947974 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.947979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.948678 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.948679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.949533 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.949535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.949910 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.949911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.950014 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.950500 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.950508 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.952765 (+ 2257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
W20260619 10:26:27.952967 18439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:27.946283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:36531 (local address 127.9.90.126:40493)
0619 10:26:27.946333 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.946335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.946527 (+ 192us) server_negotiation.cc:407] Connection header received
0619 10:26:27.947028 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.947030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.947066 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.947120 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.948082 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.948584 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.949612 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.949838 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.949902 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.952908 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:36531: BlockingRecv error: recv got EOF from 127.9.90.67:36531 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 10:26:27.953202 18700 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:40493 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.90.126:40493: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
I20260619 10:26:27.953336 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 18571
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:27.957528 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.953444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:47521)
0619 10:26:27.953484 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.953494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.953531 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.953775 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.953776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.953914 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.953957 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.953959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.954563 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.954564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.955078 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.955080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.955396 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.955396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.955461 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.955896 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.955901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.957458 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 10:26:27.957592 18439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:27.953618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:47521 (local address 127.9.90.126:40493)
0619 10:26:27.953674 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.953676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.953684 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:27.953707 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.953708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.953717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.953759 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.954021 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.954498 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.955144 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.955342 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.955392 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.957538 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:47521: BlockingRecv error: recv got EOF from 127.9.90.67:47521 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:26:27.960246 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.953664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.9.90.126:40493)
0619 10:26:27.953722 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.953724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.953901 (+ 177us) server_negotiation.cc:407] Connection header received
0619 10:26:27.953959 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.953960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.953974 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.954020 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.954546 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.955008 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.955830 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.956089 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.956155 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.957235 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:27.957240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:27.957241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:27.957253 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:27.958793 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:27.959104 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:27.959109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:27.959109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:27.959314 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:27.959536 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:27.959537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:27.959538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:27.959786 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:27.959920 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:27.960042 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 10:26:27.960174 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:27 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:27.962057 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:27.958033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:40765)
0619 10:26:27.958074 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:27.958082 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:27.958114 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:27.958243 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:27.958244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:27.958353 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:27.958378 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.958379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.958940 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.958941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.959422 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:27.959424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.959726 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.959727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:27.959774 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.960094 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:27.960098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:27.962010 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:26:27.962141 18439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:27.958080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:40765 (local address 127.9.90.126:40493)
0619 10:26:27.958142 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:27.958144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:27.958152 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:27.958170 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:27.958170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:27.958178 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:27.958231 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:27.958434 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.958879 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.959485 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:27.959671 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:27.959722 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:27.962088 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:40765: BlockingRecv error: recv got EOF from 127.9.90.67:40765 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 10:26:27.962265 18700 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 10:26:28 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:28 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:28.718398 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:28.714328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:41481)
0619 10:26:28.714441 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:28.714451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:28.714498 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:28.714634 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:28.714635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:28.714799 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:28.714846 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:28.714847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.715442 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.715442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:28.716002 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:28.716004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.716290 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.716291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:28.716327 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:28.716738 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:28.716742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:28.718333 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
W20260619 10:26:28.718484 18705 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:28.714408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:41481 (local address 127.9.90.126:40493)
0619 10:26:28.714527 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:28.714528 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:28.714536 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:28.714554 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:28.714555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:28.714564 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:28.714622 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:28.714896 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.715388 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.716061 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.716241 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.716286 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:28.718421 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:41481: BlockingRecv error: recv got EOF from 127.9.90.65:41481 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":50,"threads_started":1}
Jun 19 10:26:28 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:28 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:28.832094 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:28.827677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:49977)
0619 10:26:28.827812 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:28.827825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:28.827900 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:28.828024 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:28.828025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:28.828208 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:28.828252 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:28.828254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.828888 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.828889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:28.829423 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:28.829424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.829733 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.829734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:28.829816 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:28.830212 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:28.830216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:28.832008 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
W20260619 10:26:28.832218 18705 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:28.827759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:49977 (local address 127.9.90.126:40493)
0619 10:26:28.827817 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:28.827819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:28.827876 (+ 57us) server_negotiation.cc:407] Connection header received
0619 10:26:28.827952 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:28.827953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:28.827963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:28.828011 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:28.828321 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.828816 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.829493 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.829676 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.829729 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:28.832165 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:49977: BlockingRecv error: recv got EOF from 127.9.90.66:49977 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 10:26:28 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:28 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:28.967022 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:28.962611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:41587)
0619 10:26:28.962750 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:28.962761 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:28.962830 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:28.962954 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:28.962954 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:28.963123 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:28.963168 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:28.963170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.963835 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.963837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:28.964372 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:28.964374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.964707 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.964708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:28.964769 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:28.965204 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:28.965209 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:28.966933 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
W20260619 10:26:28.967095 18705 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:28.962689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:41587 (local address 127.9.90.126:40493)
0619 10:26:28.962750 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:28.962752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:28.962820 (+ 68us) server_negotiation.cc:407] Connection header received
0619 10:26:28.962881 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:28.962882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:28.962896 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:28.962939 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:28.963262 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.963757 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.964453 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:28.964647 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:28.964700 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:28.967043 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:41587: BlockingRecv error: recv got EOF from 127.9.90.67:41587 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 10:26:29 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:29 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:29.723372 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:29.718851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:50237)
0619 10:26:29.718972 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:29.718985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:29.719064 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:29.719187 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:29.719188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:29.719349 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:29.719410 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:29.719411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.720027 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.720028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:29.720565 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:29.720567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.720938 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.720939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:29.720989 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:29.721465 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:29.721469 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:29.723308 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
W20260619 10:26:29.723476 18709 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:29.718932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:50237 (local address 127.9.90.126:40493)
0619 10:26:29.719079 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:29.719081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:29.719089 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:29.719109 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:29.719110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:29.719119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:29.719173 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:29.719465 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.719954 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.720661 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.720862 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.720922 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:29.723397 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:50237: BlockingRecv error: recv got EOF from 127.9.90.65:50237 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
Jun 19 10:26:29 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:29 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:29.836759 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:29.832579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:54545)
0619 10:26:29.832698 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:29.832710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:29.832769 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:29.832893 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:29.832894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:29.833056 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:29.833100 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:29.833102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.833716 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.833717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:29.834277 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:29.834279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.834602 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.834603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:29.834657 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:29.835072 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:29.835076 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:29.836687 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
W20260619 10:26:29.836874 18709 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:29.832674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:54545 (local address 127.9.90.126:40493)
0619 10:26:29.832729 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:29.832731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:29.832756 (+ 25us) server_negotiation.cc:407] Connection header received
0619 10:26:29.832827 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:29.832828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:29.832838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:29.832880 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:29.833168 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.833647 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.834354 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.834544 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.834597 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:29.836810 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:54545: BlockingRecv error: recv got EOF from 127.9.90.66:54545 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 10:26:29 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:29 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:29.972054 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:29.967477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:60559)
0619 10:26:29.967622 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:29.967634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:29.967688 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:29.967868 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:29.967870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:29.968077 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:29.968130 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:29.968132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.968758 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.968760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:29.969320 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:29.969322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.969616 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.969617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:29.969664 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:29.970205 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:29.970211 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:29.971975 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":57,"threads_started":1}
W20260619 10:26:29.972117 18709 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:29.967621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:60559 (local address 127.9.90.126:40493)
0619 10:26:29.967686 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:29.967688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:29.967697 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:26:29.967749 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:29.967750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:29.967761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:29.967807 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:29.968194 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.968684 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.969386 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:29.969554 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:29.969596 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:29.972064 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:60559: BlockingRecv error: recv got EOF from 127.9.90.67:60559 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 10:26:30 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:30 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:30.727963 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:30.723780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:54391)
0619 10:26:30.723905 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:30.723916 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:30.724000 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:30.724126 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:30.724127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:30.724278 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:30.724322 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:30.724323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.724911 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.724912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:30.725470 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:30.725472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.725811 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.725811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:30.725849 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:30.726314 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:30.726318 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:30.727887 (+ 1569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260619 10:26:30.728041 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:30.723855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:54391 (local address 127.9.90.126:40493)
0619 10:26:30.723985 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:30.723987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:30.723997 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:30.724057 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:30.724057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:30.724066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:30.724120 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:30.724369 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.724850 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.725544 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.725738 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.725812 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:30.727981 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:54391: BlockingRecv error: recv got EOF from 127.9.90.65:54391 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
Jun 19 10:26:30 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:30 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:30.841624 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:30.837222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:40487)
0619 10:26:30.837334 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:30.837347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:30.837413 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:30.837546 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:30.837547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:30.837708 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:30.837754 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:30.837756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.838438 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.838439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:30.838968 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:30.838970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.839295 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.839296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:30.839348 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:30.839820 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:30.839826 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:30.841555 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
W20260619 10:26:30.841706 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:30.837318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:40487 (local address 127.9.90.126:40493)
0619 10:26:30.837380 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:30.837382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:30.837395 (+ 13us) server_negotiation.cc:407] Connection header received
0619 10:26:30.837475 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:30.837476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:30.837487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:30.837530 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:30.837871 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.838366 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.839046 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.839237 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.839291 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:30.841656 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:40487: BlockingRecv error: recv got EOF from 127.9.90.66:40487 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 10:26:30 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:30 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:30.977100 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:30.972545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:50853)
0619 10:26:30.972668 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:30.972682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:30.972765 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:30.972891 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:30.972892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:30.973064 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:30.973110 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:30.973112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.973843 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.973844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:30.974428 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:30.974431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.974775 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.974777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:30.974831 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:30.975257 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:30.975261 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:30.977024 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
W20260619 10:26:30.977190 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:30.972644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:50853 (local address 127.9.90.126:40493)
0619 10:26:30.972726 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:30.972728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:30.972739 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:26:30.972829 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:30.972830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:30.972842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:30.972885 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:30.973185 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.973736 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.974504 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:30.974717 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:30.974772 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:30.977122 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:50853: BlockingRecv error: recv got EOF from 127.9.90.67:50853 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 10:26:31 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:31 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:31.733114 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:31.728453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:41135)
0619 10:26:31.728574 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:31.728584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:31.728642 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:31.728775 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:31.728775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:31.728928 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:31.728974 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:31.728976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.729675 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.729676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:31.730228 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:31.730230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.730611 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.730612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:31.730669 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:31.731243 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:31.731247 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:31.733054 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20260619 10:26:31.733201 18717 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:31.728549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:41135 (local address 127.9.90.126:40493)
0619 10:26:31.728656 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:31.728658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:31.728681 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:26:31.728702 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:31.728703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:31.728712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:31.728760 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:31.729085 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.729584 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.730294 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.730528 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.730597 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:31.733132 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:41135: BlockingRecv error: recv got EOF from 127.9.90.65:41135 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
Jun 19 10:26:31 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:31 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:31.846164 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:31.842157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:49071)
0619 10:26:31.842275 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:31.842285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:31.842352 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:31.842462 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:31.842462 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:31.842612 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:31.842658 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:31.842659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.843280 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.843281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:31.843818 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:31.843820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.844114 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.844115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:31.844149 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:31.844549 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:31.844553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:31.846109 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
W20260619 10:26:31.846273 18717 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:31.842217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:49071 (local address 127.9.90.126:40493)
0619 10:26:31.842284 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:31.842286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:31.842343 (+ 57us) server_negotiation.cc:407] Connection header received
0619 10:26:31.842395 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:31.842395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:31.842405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:31.842449 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:31.842708 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.843217 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.843882 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.844062 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.844111 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:31.846193 (+ 2082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:49071: BlockingRecv error: recv got EOF from 127.9.90.66:49071 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 10:26:31 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:31 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:31.981813 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:31.977534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:38407)
0619 10:26:31.977716 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:31.977727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:31.977804 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:31.977926 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:31.977927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:31.978075 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:31.978121 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:31.978123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.978717 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.978717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:31.979231 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:31.979233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.979574 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.979574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:31.979611 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:31.980050 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:31.980055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:31.981692 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":123,"threads_started":1}
W20260619 10:26:31.981896 18717 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:31.977588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:38407 (local address 127.9.90.126:40493)
0619 10:26:31.977693 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:31.977695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:31.977809 (+ 114us) server_negotiation.cc:407] Connection header received
0619 10:26:31.977861 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:31.977862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:31.977878 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:31.977915 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:31.978166 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.978652 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.979320 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:31.979512 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:31.979571 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:31.981825 (+ 2254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:38407: BlockingRecv error: recv got EOF from 127.9.90.67:38407 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 10:26:32 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:32 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:32.742228 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:32.733607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:47191)
0619 10:26:32.733750 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:32.733760 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:32.733844 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:32.734018 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:32.734019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:32.734184 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:32.734245 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:32.734247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.734891 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.734892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:32.735424 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:32.735425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.735708 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.735709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:32.735750 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:32.740403 (+ 4653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:32.740409 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:32.742133 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
W20260619 10:26:32.742337 18721 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:32.733682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:47191 (local address 127.9.90.126:40493)
0619 10:26:32.733813 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:32.733815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:32.733827 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:26:32.733899 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:32.733900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:32.733909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:32.733970 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:32.734311 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.734807 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.735485 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.735662 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.735717 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:32.742259 (+ 6542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:47191: BlockingRecv error: recv got EOF from 127.9.90.65:47191 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
Jun 19 10:26:32 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:32 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:32.850769 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:32.846580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:50255)
0619 10:26:32.846738 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:32.846749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:32.846815 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:32.846936 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:32.846936 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:32.847102 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:32.847147 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:32.847148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.847767 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.847769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:32.848302 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:32.848304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.848617 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.848617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:32.848670 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:32.849082 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:32.849087 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:32.850685 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260619 10:26:32.850835 18721 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:32.846720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:50255 (local address 127.9.90.126:40493)
0619 10:26:32.846768 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:32.846770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:32.846798 (+ 28us) server_negotiation.cc:407] Connection header received
0619 10:26:32.846865 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:32.846866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:32.846877 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:32.846920 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:32.847213 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.847692 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.848378 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.848560 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.848612 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:32.850785 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:50255: BlockingRecv error: recv got EOF from 127.9.90.66:50255 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 19 10:26:32 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:32 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:32.986827 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:32.982314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:38421)
0619 10:26:32.982450 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:32.982462 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:32.982533 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:32.982650 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:32.982650 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:32.982815 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:32.982860 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:32.982862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.983486 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.983486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:32.983999 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:32.984001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.984342 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.984342 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:32.984397 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:32.984892 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:32.984898 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:32.986767 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
W20260619 10:26:32.986980 18721 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:32.982388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:38421 (local address 127.9.90.126:40493)
0619 10:26:32.982454 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:32.982456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:32.982525 (+ 69us) server_negotiation.cc:407] Connection header received
0619 10:26:32.982584 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:32.982585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:32.982598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:32.982645 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:32.982931 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.983415 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.984081 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:32.984281 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:32.984335 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:32.986910 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:38421: BlockingRecv error: recv got EOF from 127.9.90.67:38421 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 10:26:33 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:33 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:33.746806 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:33.742660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:47783)
0619 10:26:33.742798 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:33.742810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:33.742862 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:33.742981 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:33.742982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:33.743130 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:33.743176 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:33.743178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.743798 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.743799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:33.744372 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:33.744375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.744703 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.744704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:33.744739 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:33.745163 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:33.745167 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:33.746749 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
W20260619 10:26:33.746907 18725 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:33.742746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:47783 (local address 127.9.90.126:40493)
0619 10:26:33.742867 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:33.742869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:33.742877 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:33.742906 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:33.742906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:33.742915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:33.742972 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:33.743250 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.743739 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.744444 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.744648 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.744699 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:33.746839 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:47783: BlockingRecv error: recv got EOF from 127.9.90.65:47783 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
Jun 19 10:26:33 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:33 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:33.855742 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:33.851264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:56563)
0619 10:26:33.851411 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:33.851421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:33.851499 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:33.851658 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:33.851659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:33.851827 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:33.851871 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:33.851872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.852518 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.852519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:33.853054 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:33.853056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.853419 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.853421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:33.853482 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:33.853994 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:33.853998 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:33.855684 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
W20260619 10:26:33.855840 18725 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:33.851344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:56563 (local address 127.9.90.126:40493)
0619 10:26:33.851394 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:33.851396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:33.851487 (+ 91us) server_negotiation.cc:407] Connection header received
0619 10:26:33.851555 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:33.851556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:33.851567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:33.851603 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:33.851931 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.852435 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.853134 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.853337 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.853389 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:33.855782 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:56563: BlockingRecv error: recv got EOF from 127.9.90.66:56563 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 19 10:26:33 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:33 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:33.991266 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:33.987233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:51903)
0619 10:26:33.987351 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:33.987360 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:33.987415 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:33.987516 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:33.987516 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:33.987665 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:33.987713 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:33.987715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.988298 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.988299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:33.988810 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:33.988811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.989077 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.989078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:33.989112 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:33.989517 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:33.989521 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:33.991187 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
W20260619 10:26:33.991400 18725 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:33.987290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:51903 (local address 127.9.90.126:40493)
0619 10:26:33.987335 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:33.987337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:33.987404 (+ 67us) server_negotiation.cc:407] Connection header received
0619 10:26:33.987452 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:33.987453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:33.987463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:33.987504 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:33.987763 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.988244 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.988862 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:33.989031 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:33.989072 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:33.991349 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:51903: BlockingRecv error: recv got EOF from 127.9.90.67:51903 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 10:26:34 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:34 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:34.751543 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:34.747227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:58927)
0619 10:26:34.747388 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:34.747399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:34.747453 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:34.747580 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:34.747580 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:34.747737 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:34.747783 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:34.747784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.748373 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.748374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:34.748890 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:34.748892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.749207 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.749207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:34.749250 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:34.749694 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:34.749698 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:34.751485 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
W20260619 10:26:34.751652 18729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:34.747261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:58927 (local address 127.9.90.126:40493)
0619 10:26:34.747365 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:34.747367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:34.747440 (+ 73us) server_negotiation.cc:407] Connection header received
0619 10:26:34.747505 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:34.747506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:34.747515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:34.747570 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:34.747832 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.748321 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.748950 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.749151 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.749199 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:34.751580 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:58927: BlockingRecv error: recv got EOF from 127.9.90.65:58927 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":57,"threads_started":1}
Jun 19 10:26:34 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:34 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:34.860522 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:34.856213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:52689)
0619 10:26:34.856378 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:34.856389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:34.856455 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:34.856575 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:34.856576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:34.856750 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:34.856797 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:34.856799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.857438 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.857439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:34.858024 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:34.858026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.858340 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.858341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:34.858397 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:34.858834 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:34.858838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:34.860446 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
W20260619 10:26:34.860598 18729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:34.856286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:52689 (local address 127.9.90.126:40493)
0619 10:26:34.856347 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:34.856349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:34.856448 (+ 99us) server_negotiation.cc:407] Connection header received
0619 10:26:34.856508 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:34.856509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:34.856520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:34.856564 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:34.856869 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.857364 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.858093 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.858282 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.858336 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:34.860548 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:52689: BlockingRecv error: recv got EOF from 127.9.90.66:52689 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 19 10:26:34 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:34 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:34.995982 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:34.991735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:34787)
0619 10:26:34.991876 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:34.991889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:34.991958 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:34.992074 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:34.992075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:34.992236 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:34.992285 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:34.992286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.992891 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.992892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:34.993432 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:34.993433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.993779 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.993780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:34.993868 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:34.994337 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:34.994343 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:34.995924 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
W20260619 10:26:34.996068 18729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:34.991844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:34787 (local address 127.9.90.126:40493)
0619 10:26:34.991902 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:34.991904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:34.991942 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:26:34.992010 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:34.992011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:34.992024 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:34.992061 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:34.992337 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.992830 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.993489 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:34.993706 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:34.993767 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:34.996006 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:34787: BlockingRecv error: recv got EOF from 127.9.90.67:34787 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 10:26:35 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:35 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:35.756640 18732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:35.752094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:58429)
0619 10:26:35.752253 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:35.752267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:35.752337 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:35.752521 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:35.752523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:35.752677 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:35.752725 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:35.752726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.753389 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.753390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:35.754141 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:35.754143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.754445 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.754446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:35.754496 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:35.754896 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:35.754901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:35.756583 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
W20260619 10:26:35.756757 18733 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:35.752247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:58429 (local address 127.9.90.126:40493)
0619 10:26:35.752372 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:35.752374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:35.752381 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:35.752402 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:35.752403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:35.752411 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:35.752474 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:35.752824 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.753321 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.754199 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.754392 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.754441 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:35.756679 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:58429: BlockingRecv error: recv got EOF from 127.9.90.65:58429 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
Jun 19 10:26:35 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:35 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:35.865564 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:35.860978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:37065)
0619 10:26:35.861120 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:35.861130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:35.861217 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:35.861383 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:35.861385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:35.861573 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:35.861619 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:35.861622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.862452 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.862453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:35.863033 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:35.863036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.863369 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.863370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:35.863426 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:35.863823 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:35.863827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:35.865488 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
W20260619 10:26:35.865641 18733 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:35.861075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:37065 (local address 127.9.90.126:40493)
0619 10:26:35.861137 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:35.861139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:35.861199 (+ 60us) server_negotiation.cc:407] Connection header received
0619 10:26:35.861301 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:35.861302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:35.861313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:35.861362 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:35.861697 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.862382 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.863115 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.863314 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.863364 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:35.865590 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:37065: BlockingRecv error: recv got EOF from 127.9.90.66:37065 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 10:26:35 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:35 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:36.000655 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:35.996448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:57701)
0619 10:26:35.996567 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:35.996578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:35.996637 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:35.996765 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:35.996765 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:35.996912 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:35.996958 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:35.996960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.997623 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.997624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:35.998228 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:35.998231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.998521 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.998522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:35.998557 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:35.998925 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:35.998929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:36.000580 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
W20260619 10:26:36.000725 18733 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:35.996521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:57701 (local address 127.9.90.126:40493)
0619 10:26:35.996581 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:35.996583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:35.996622 (+ 39us) server_negotiation.cc:407] Connection header received
0619 10:26:35.996695 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:35.996696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:35.996707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:35.996753 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:35.997025 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.997558 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.998291 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:35.998470 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:35.998518 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:36.000671 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:57701: BlockingRecv error: recv got EOF from 127.9.90.67:57701 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 10:26:36 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:36 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:36.761052 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:36.757057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:35983)
0619 10:26:36.757197 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:36.757208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:36.757260 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:36.757377 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:36.757377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:36.757528 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:36.757574 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:36.757575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:36.758192 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:36.758192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:36.758712 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:36.758714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:36.758996 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:36.758997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:36.759031 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:36.759417 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:36.759421 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:36.760993 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
W20260619 10:26:36.761149 18737 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:36.757128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:35983 (local address 127.9.90.126:40493)
0619 10:26:36.757238 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:36.757240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:36.757250 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:36.757306 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:36.757307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:36.757316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:36.757366 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:36.757616 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:36.758136 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:36.758768 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:36.758951 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:36.758991 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:36.761080 (+ 2089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:35983: BlockingRecv error: recv got EOF from 127.9.90.65:35983 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
Jun 19 10:26:36 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:36 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:36.870752 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:36.866085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:50793)
0619 10:26:36.866215 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:36.866227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:36.866287 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:36.866395 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:36.866395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:36.866546 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:36.866591 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:36.866592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:36.867212 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:36.867213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:36.867755 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:36.867757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:36.868059 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:36.868060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:36.868097 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:36.868511 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:36.868516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:36.870663 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
W20260619 10:26:36.870792 18737 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:36.866140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:50793 (local address 127.9.90.126:40493)
0619 10:26:36.866214 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:36.866216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:36.866279 (+ 63us) server_negotiation.cc:407] Connection header received
0619 10:26:36.866328 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:36.866329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:36.866339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:36.866385 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:36.866637 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:36.867116 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:36.867819 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:36.868005 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:36.868056 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:36.870742 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:50793: BlockingRecv error: recv got EOF from 127.9.90.66:50793 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 10:26:36 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:36 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:37.005757 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:37.001098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:44253)
0619 10:26:37.001222 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:37.001235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:37.001309 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:37.001441 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:37.001442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:37.001610 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:37.001657 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:37.001658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.002362 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.002363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:37.002889 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:37.002891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.003223 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.003224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:37.003279 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:37.003671 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:37.003675 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:37.005654 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
W20260619 10:26:37.005877 18737 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:37.001168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:44253 (local address 127.9.90.126:40493)
0619 10:26:37.001226 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:37.001229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:37.001297 (+ 68us) server_negotiation.cc:407] Connection header received
0619 10:26:37.001367 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:37.001368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:37.001382 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:37.001429 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:37.001742 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.002287 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.002959 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.003143 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.003203 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:37.005806 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:44253: BlockingRecv error: recv got EOF from 127.9.90.67:44253 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 10:26:37 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:37 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:37.768867 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:37.761495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:34903)
0619 10:26:37.761607 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:37.761620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:37.761702 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:37.761879 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:37.761880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:37.762033 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:37.762081 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:37.762083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.762750 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.762752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:37.763276 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:37.763278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.763604 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.763605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:37.763640 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:37.767091 (+ 3451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:37.767095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:37.768796 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
W20260619 10:26:37.768944 18741 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:37.761587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:34903 (local address 127.9.90.126:40493)
0619 10:26:37.761753 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:37.761755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:37.761763 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:37.761798 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:37.761799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:37.761809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:37.761861 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:37.762168 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.762677 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.763351 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.763548 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.763601 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:37.768883 (+ 5282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:34903: BlockingRecv error: recv got EOF from 127.9.90.65:34903 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
Jun 19 10:26:37 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:37 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:37.876339 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:37.871192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:56127)
0619 10:26:37.871317 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:37.871330 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:37.871393 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:37.871504 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:37.871505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:37.871650 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:37.871700 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:37.871702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.872441 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.872442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:37.873229 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:37.873232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.873713 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.873714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:37.873778 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:37.874350 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:37.874369 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:37.876277 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
W20260619 10:26:37.876506 18741 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:37.871274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:56127 (local address 127.9.90.126:40493)
0619 10:26:37.871337 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:37.871339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:37.871377 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:26:37.871435 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:37.871436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:37.871446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:37.871492 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:37.871782 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.872356 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.873325 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:37.873615 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:37.873697 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:37.876442 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:56127: BlockingRecv error: recv got EOF from 127.9.90.66:56127 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 10:26:38 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:38 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:38.010946 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:38.006408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:51163)
0619 10:26:38.006561 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:38.006574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:38.006631 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:38.006758 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:38.006759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:38.006909 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:38.006963 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:38.006965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.007601 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.007602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:38.008128 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:38.008130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.008485 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.008486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:38.008522 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:38.008967 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:38.008973 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:38.010884 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
W20260619 10:26:38.011041 18741 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:38.006523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:51163 (local address 127.9.90.126:40493)
0619 10:26:38.006581 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:38.006583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:38.006617 (+ 34us) server_negotiation.cc:407] Connection header received
0619 10:26:38.006695 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:38.006696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:38.006709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:38.006745 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:38.007017 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.007534 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.008224 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.008419 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.008478 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:38.010973 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:51163: BlockingRecv error: recv got EOF from 127.9.90.67:51163 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 19 10:26:38 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:38 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:38.773736 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:38.769337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:57037)
0619 10:26:38.769477 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:38.769487 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:38.769541 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:38.769697 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:38.769698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:38.769881 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:38.769923 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:38.769925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.770591 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.770592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:38.771145 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:38.771147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.771470 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.771471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:38.771525 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:38.771919 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:38.771923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:38.773659 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
W20260619 10:26:38.773840 18745 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:38.769436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:57037 (local address 127.9.90.126:40493)
0619 10:26:38.769545 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:38.769547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:38.769575 (+ 28us) server_negotiation.cc:407] Connection header received
0619 10:26:38.769594 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:38.769594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:38.769604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:38.769652 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:38.769989 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.770508 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.771217 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.771399 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.771443 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:38.773745 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:57037: BlockingRecv error: recv got EOF from 127.9.90.65:57037 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
Jun 19 10:26:38 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:38 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:38.882161 18746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:38.876786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:34865)
0619 10:26:38.877132 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:38.877166 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:38.877241 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:38.877446 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:38.877447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:38.877643 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:38.877701 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:38.877704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.878640 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.878641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:38.879304 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:38.879306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.879665 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.879666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:38.879720 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:38.880187 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:38.880192 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:38.882062 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":296,"threads_started":1}
W20260619 10:26:38.882221 18745 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:38.876821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:34865 (local address 127.9.90.126:40493)
0619 10:26:38.876887 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:38.876889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:38.877239 (+ 350us) server_negotiation.cc:407] Connection header received
0619 10:26:38.877329 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:38.877330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:38.877347 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:38.877418 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:38.877781 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.878549 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.879379 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:38.879593 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:38.879657 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:38.882167 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:34865: BlockingRecv error: recv got EOF from 127.9.90.66:34865 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 10:26:39 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:39 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:39.017019 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:39.011433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:44483)
0619 10:26:39.012150 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:39.012163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:39.012241 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:39.012390 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:39.012391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:39.012608 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:39.012665 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:39.012669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.013556 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.013558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:39.014378 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:39.014382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.014746 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.014747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:39.014787 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:39.015246 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:39.015251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:39.016893 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":63,"threads_started":1}
W20260619 10:26:39.017023 18745 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:39.011587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:44483 (local address 127.9.90.126:40493)
0619 10:26:39.011639 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:39.011642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:39.012224 (+ 582us) server_negotiation.cc:407] Connection header received
0619 10:26:39.012295 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:39.012296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:39.012312 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:39.012372 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:39.012736 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.013469 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.014456 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.014686 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.014741 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:39.016975 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:44483: BlockingRecv error: recv got EOF from 127.9.90.67:44483 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 19 10:26:39 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:39 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:39.778949 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:39.774166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:53187)
0619 10:26:39.774284 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:39.774294 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:39.774341 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:39.774464 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:39.774465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:39.774628 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:39.774677 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:39.774678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.775511 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.775527 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:39.776054 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:39.776056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.776508 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.776509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:39.776577 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:39.777050 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:39.777056 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:39.778879 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
W20260619 10:26:39.779017 18749 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:39.774225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:53187 (local address 127.9.90.126:40493)
0619 10:26:39.774340 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:39.774342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:39.774374 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:26:39.774393 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:39.774394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:39.774403 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:39.774454 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:39.774749 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.775449 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.776185 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.776425 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.776474 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:39.778961 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:53187: BlockingRecv error: recv got EOF from 127.9.90.65:53187 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
Jun 19 10:26:39 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:39 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:39.886811 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:39.882620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:46259)
0619 10:26:39.882770 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:39.882780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:39.882833 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:39.882954 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:39.882955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:39.883104 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:39.883150 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:39.883151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.883809 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.883810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:39.884330 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:39.884331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.884678 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.884678 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:39.884714 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:39.885127 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:39.885130 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:39.886741 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
W20260619 10:26:39.886974 18749 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:39.882706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:46259 (local address 127.9.90.126:40493)
0619 10:26:39.882749 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:39.882751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:39.882819 (+ 68us) server_negotiation.cc:407] Connection header received
0619 10:26:39.882886 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:39.882887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:39.882904 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:39.882943 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:39.883237 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.883741 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.884410 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:39.884618 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:39.884679 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:39.886864 (+ 2185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:46259: BlockingRecv error: recv got EOF from 127.9.90.66:46259 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 19 10:26:40 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:40 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:40.022867 18751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:40.017608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:46933)
0619 10:26:40.017779 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:40.017812 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:40.017892 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:40.018030 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:40.018031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:40.018229 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:40.018275 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:40.018278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.018946 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.018946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:40.019552 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:40.019555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.020134 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.020135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:40.020204 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:40.020718 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:40.020724 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:40.022787 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
W20260619 10:26:40.022996 18749 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:40.017781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:46933 (local address 127.9.90.126:40493)
0619 10:26:40.017868 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:40.017870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:40.017880 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:26:40.017959 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:40.017960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:40.017974 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:40.018024 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:40.018358 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.018861 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.019790 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.019986 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.020062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:40.022920 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:46933: BlockingRecv error: recv got EOF from 127.9.90.67:46933 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 10:26:40 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:40 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:40.783741 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:40.779445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:49501)
0619 10:26:40.779565 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:40.779577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:40.779630 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:40.779783 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:40.779784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:40.779978 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:40.780028 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:40.780030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.780627 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.780628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:40.781184 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:40.781186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.781490 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.781491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:40.781536 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:40.781984 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:40.781988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:40.783673 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
W20260619 10:26:40.783823 18753 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:40.779503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:49501 (local address 127.9.90.126:40493)
0619 10:26:40.779648 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:40.779650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:40.779665 (+ 15us) server_negotiation.cc:407] Connection header received
0619 10:26:40.779689 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:40.779690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:40.779700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:40.779767 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:40.780088 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.780573 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.781247 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.781427 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.781470 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:40.783760 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:49501: BlockingRecv error: recv got EOF from 127.9.90.65:49501 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
Jun 19 10:26:40 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:40 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:40.891419 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:40.887234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:36661)
0619 10:26:40.887413 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:40.887422 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:40.887476 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:40.887592 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:40.887593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:40.887749 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:40.887795 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:40.887797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.888481 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.888482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:40.889004 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:40.889005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.889371 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.889372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:40.889410 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:40.889780 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:40.889804 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:40.891361 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
W20260619 10:26:40.891510 18753 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:40.887350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:36661 (local address 127.9.90.126:40493)
0619 10:26:40.887413 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:40.887415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:40.887485 (+ 70us) server_negotiation.cc:407] Connection header received
0619 10:26:40.887516 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:40.887516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:40.887527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:40.887580 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:40.887880 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.888400 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.889063 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:40.889304 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:40.889368 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:40.891452 (+ 2084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:36661: BlockingRecv error: recv got EOF from 127.9.90.66:36661 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 10:26:41 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:41 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:41.027781 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:41.023357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:43431)
0619 10:26:41.023507 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:41.023519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:41.023594 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:41.023720 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:41.023721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:41.023929 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:41.023992 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:41.023995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.024629 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.024630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:41.025272 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:41.025274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.025596 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.025597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:41.025650 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:41.026078 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:41.026083 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:41.027706 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
W20260619 10:26:41.027856 18753 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:41.023445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:43431 (local address 127.9.90.126:40493)
0619 10:26:41.023514 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:41.023516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:41.023581 (+ 65us) server_negotiation.cc:407] Connection header received
0619 10:26:41.023647 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:41.023648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:41.023661 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:41.023710 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:41.024054 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.024559 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.025350 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.025539 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.025590 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:41.027799 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:43431: BlockingRecv error: recv got EOF from 127.9.90.67:43431 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 10:26:41 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:41 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:41.788478 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:41.784165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:38147)
0619 10:26:41.784286 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:41.784295 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:41.784344 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:41.784465 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:41.784465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:41.784620 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:41.784673 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:41.784675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.785390 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.785390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:41.785970 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:41.785972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.786352 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.786352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:41.786387 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:41.786843 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:41.786847 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:41.788417 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260619 10:26:41.788583 18757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:41.784255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:38147 (local address 127.9.90.126:40493)
0619 10:26:41.784360 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:41.784362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:41.784369 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:26:41.784389 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:41.784390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:41.784398 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:41.784464 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:41.784769 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.785319 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.786079 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.786288 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.786351 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:41.788508 (+ 2157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:38147: BlockingRecv error: recv got EOF from 127.9.90.65:38147 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
Jun 19 10:26:41 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:41 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:41.895907 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:41.891857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:33031)
0619 10:26:41.891960 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:41.891973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:41.892041 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:41.892164 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:41.892165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:41.892330 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:41.892373 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:41.892375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.892995 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.892996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:41.893555 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:41.893556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.893921 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.893923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:41.893975 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:41.894356 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:41.894360 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:41.895820 (+ 1460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
W20260619 10:26:41.895962 18757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:41.891940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:33031 (local address 127.9.90.126:40493)
0619 10:26:41.892036 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:41.892037 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:41.892046 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:26:41.892102 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:41.892103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:41.892114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:41.892155 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:41.892442 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.892926 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.893630 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:41.893866 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:41.893921 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:41.895913 (+ 1992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:33031: BlockingRecv error: recv got EOF from 127.9.90.66:33031 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 10:26:42 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:42 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.67@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:42.033205 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:42.028298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.67:44781)
0619 10:26:42.028442 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:42.028454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:42.028517 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:42.028642 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:42.028643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:42.028798 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:42.028841 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:42.028842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.029607 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.029609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:42.030329 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:42.030332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.030689 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.030690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:42.030744 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:42.031203 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:42.031207 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:42.033126 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
W20260619 10:26:42.033319 18757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:42.028397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:44781 (local address 127.9.90.126:40493)
0619 10:26:42.028449 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:42.028450 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:42.028509 (+ 59us) server_negotiation.cc:407] Connection header received
0619 10:26:42.028559 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:42.028560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:42.028572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:42.028620 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:42.028955 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.029513 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.030414 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.030616 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.030695 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:42.033260 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.67:44781: BlockingRecv error: recv got EOF from 127.9.90.67:44781 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 10:26:42 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:42 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.65@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:42.796432 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:42.788962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.65:33721)
0619 10:26:42.789100 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:42.789109 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:42.789153 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:42.789285 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:42.789286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:42.789442 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:42.789487 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:42.789488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.790129 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.790130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:42.790663 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:42.790665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.790959 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.790960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:42.791000 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:42.794552 (+ 3552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:42.794558 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:42.796357 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260619 10:26:42.796526 18761 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:42.789066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:33721 (local address 127.9.90.126:40493)
0619 10:26:42.789177 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:42.789179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:42.789187 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:42.789205 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:42.789206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:42.789216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:42.789273 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:42.789541 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.790059 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.790727 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.790908 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.790947 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:42.796456 (+ 5509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.65:33721: BlockingRecv error: recv got EOF from 127.9.90.65:33721 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":50,"threads_started":1}
Jun 19 10:26:42 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 10:26:42 dist-test-slave-hw7q krb5kdc[17703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.90.66@KRBTEST.COM for oryx/127.9.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 10:26:42.900655 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:42.896415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.9.90.66:39815)
0619 10:26:42.896549 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:42.896559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:42.896617 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:42.896766 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:42.896767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:42.896916 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:42.896964 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:42.896965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.897675 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.897676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:42.898270 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:42.898272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.898586 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.898587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:42.898626 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:42.899013 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:42.899017 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:42.900596 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.90.126:40493: Server oryx/127.9.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
W20260619 10:26:42.900718 18761 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:42.896519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:39815 (local address 127.9.90.126:40493)
0619 10:26:42.896561 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:42.896563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:42.896601 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:26:42.896659 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:42.896660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:42.896670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:42.896715 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:42.897059 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.897601 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.898341 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:42.898536 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:42.898580 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:42.900670 (+ 2090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.90.66:39815: BlockingRecv error: recv got EOF from 127.9.90.66:39815 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:42.959247 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 18286
I20260619 10:26:42.963409 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 18438
I20260619 10:26:42.967118 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 18571
I20260619 10:26:42.970403 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 18216
2026-06-19T10:26:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17288 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T10:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:42Z Disabled control of system clock
I20260619 10:26:42.996153 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42641
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:42641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 10:26:43.075695 18766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:43.075892 18766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:43.075911 18766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:43.077092 18766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:43.077131 18766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:43.077140 18766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:43.077152 18766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:43.077162 18766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:43.078505 18766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:42641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:43.078724 18766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:43.079005 18766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:43.083487 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.083459 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.083487 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.083559 18766 server_base.cc:1034] running on GCE node
I20260619 10:26:43.083871 18766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:43.084106 18766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:43.085286 18766 hybrid_clock.cc:648] HybridClock initialized: now 1781864803085273 us; error 30 us; skew 500 ppm
I20260619 10:26:43.086592 18766 webserver.cc:460] Webserver started at http://127.9.90.126:33099/ using document root <none> and password file <none>
I20260619 10:26:43.086820 18766 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:43.086864 18766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:43.086995 18766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:43.088011 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "16f1abb9e55a4e5787759b9e52c8731e"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "36ddf68de83a7e91e120760ee161850b"
server_key_iv: "29b318bae3af33b3652627307b2b940a"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.088323 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "16f1abb9e55a4e5787759b9e52c8731e"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "36ddf68de83a7e91e120760ee161850b"
server_key_iv: "29b318bae3af33b3652627307b2b940a"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.089510 18766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:43.090209 18780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.090355 18766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.090435 18766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "16f1abb9e55a4e5787759b9e52c8731e"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "36ddf68de83a7e91e120760ee161850b"
server_key_iv: "29b318bae3af33b3652627307b2b940a"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.090487 18766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:43.108791 18766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:43.109047 18766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:43.109146 18766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:43.112573 18766 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:42641
I20260619 10:26:43.112607 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:42641 every 8 connection(s)
I20260619 10:26:43.112927 18766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:43.113554 18833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:43.116092 18833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e: Bootstrap starting.
I20260619 10:26:43.116649 18833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:43.116902 18833 log.cc:824] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:43.117504 18833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e: No bootstrap required, opened a new log
I20260619 10:26:43.119071 18833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f1abb9e55a4e5787759b9e52c8731e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42641 } }
I20260619 10:26:43.119212 18833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:43.119234 18833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f1abb9e55a4e5787759b9e52c8731e, State: Initialized, Role: FOLLOWER
I20260619 10:26:43.119364 18833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f1abb9e55a4e5787759b9e52c8731e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42641 } }
I20260619 10:26:43.119436 18833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:43.119468 18833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:43.119513 18833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:43.120010 18833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f1abb9e55a4e5787759b9e52c8731e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42641 } }
I20260619 10:26:43.120092 18833 leader_election.cc:302] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16f1abb9e55a4e5787759b9e52c8731e; no voters:
I20260619 10:26:43.120234 18833 leader_election.cc:288] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:43.120291 18836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:43.120466 18833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:43.120465 18836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [term 1 LEADER]: Becoming Leader. State: Replica: 16f1abb9e55a4e5787759b9e52c8731e, State: Running, Role: LEADER
I20260619 10:26:43.120656 18836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f1abb9e55a4e5787759b9e52c8731e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42641 } }
I20260619 10:26:43.120677 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 18766
I20260619 10:26:43.120754 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:43.120950 9577 external_mini_cluster.cc:1428] Setting key 1cf7dca7c21054bbcb0a5c24cb4baf21
I20260619 10:26:43.121233 18838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16f1abb9e55a4e5787759b9e52c8731e. Latest consensus state: current_term: 1 leader_uuid: "16f1abb9e55a4e5787759b9e52c8731e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f1abb9e55a4e5787759b9e52c8731e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42641 } } }
I20260619 10:26:43.121318 18838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:43.121587 18837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16f1abb9e55a4e5787759b9e52c8731e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f1abb9e55a4e5787759b9e52c8731e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42641 } } }
I20260619 10:26:43.121665 18837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:43.121778 18845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:43.122277 18845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:43.124118 18845 catalog_manager.cc:1347] Generated new cluster ID: d426a9dd239b4ddc8504c7e624659585
I20260619 10:26:43.124161 18845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:43.127061 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.121464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.9.90.126:42641)
0619 10:26:43.122268 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:43.122277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:43.122307 (+ 30us) server_negotiation.cc:407] Connection header received
0619 10:26:43.122494 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:43.122501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:43.122569 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:43.122710 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:43.122921 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.124014 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.125069 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.125675 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.125803 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:43.126412 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:43.126416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:43.126416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:43.126604 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:43.126643 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:43.126656 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:43.126713 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:43.126792 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 10:26:43.126864 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":63,"threads_started":1}
I20260619 10:26:43.147796 18845 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:43.148317 18845 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:43.151288 18845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16f1abb9e55a4e5787759b9e52c8731e: Generated new TSK 0
I20260619 10:26:43.151443 18845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:26:43.154903 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42641
--builtin_ntp_servers=127.9.90.84:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 10:26:43.233436 18857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:43.233631 18857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:43.233651 18857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:43.234817 18857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:43.234859 18857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:43.234882 18857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:43.236158 18857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:43.236363 18857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:43.236634 18857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:43.241827 18862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.241878 18863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.241998 18857 server_base.cc:1034] running on GCE node
W20260619 10:26:43.241878 18865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.242235 18857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:43.242461 18857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:43.243610 18857 hybrid_clock.cc:648] HybridClock initialized: now 1781864803243593 us; error 32 us; skew 500 ppm
I20260619 10:26:43.244766 18857 webserver.cc:460] Webserver started at http://127.9.90.65:40521/ using document root <none> and password file <none>
I20260619 10:26:43.245052 18857 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:43.245110 18857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:43.245234 18857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:43.246304 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "b061266ab1c34c338e02bc53c0b07936"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "a7ec0864f0462a9d8b9492569c818f87"
server_key_iv: "a46271ad969d9c05e5aad5cf4abcf9b2"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.246645 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "b061266ab1c34c338e02bc53c0b07936"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "a7ec0864f0462a9d8b9492569c818f87"
server_key_iv: "a46271ad969d9c05e5aad5cf4abcf9b2"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.247818 18857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:43.248477 18871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.248651 18857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.248731 18857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "b061266ab1c34c338e02bc53c0b07936"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "a7ec0864f0462a9d8b9492569c818f87"
server_key_iv: "a46271ad969d9c05e5aad5cf4abcf9b2"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.248803 18857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:43.257068 18857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:43.257292 18857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:43.257381 18857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:43.257545 18857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:43.257826 18857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:43.257865 18857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.257893 18857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:43.257910 18857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.263100 18857 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:34047
I20260619 10:26:43.263164 18984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:34047 every 8 connection(s)
I20260619 10:26:43.263420 18857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:43.267947 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.264038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:60221 (local address 127.9.90.126:42641)
0619 10:26:43.264096 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:43.264098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:43.264302 (+ 204us) server_negotiation.cc:407] Connection header received
0619 10:26:43.264750 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:43.264751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:43.264764 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:43.264813 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:43.265391 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.265930 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.266812 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.267020 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.267063 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:43.267765 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:43.267766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:43.267766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:43.267770 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:43.267780 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:43.267783 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:43.267818 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:43.267871 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 10:26:43.267892 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:26:43.268110 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.264026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42641 (local address 127.9.90.65:60221)
0619 10:26:43.264201 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:43.264237 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:43.264678 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:43.264863 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:43.264866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:43.264874 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:43.265331 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:43.265335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.265999 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.265999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:43.266756 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:43.266758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.267073 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.267073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:43.267186 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:43.267656 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:43.267657 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:43.267683 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:43.267705 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:43.267820 (+ 115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:43.267822 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:43.267824 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:26:43.267866 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:26:43.267935 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260619 10:26:43.268556 18985 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42641
I20260619 10:26:43.268632 18985 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:43.268788 18985 heartbeater.cc:511] Master 127.9.90.126:42641 requested a full tablet report, sending...
I20260619 10:26:43.269169 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 18857
I20260619 10:26:43.269241 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:43.269227 18797 ts_manager.cc:194] Registered new tserver with Master: b061266ab1c34c338e02bc53c0b07936 (127.9.90.65:34047)
I20260619 10:26:43.269418 9577 external_mini_cluster.cc:1428] Setting key 8dc6224eda6c00b7a1beb87cb6aba5ad
I20260619 10:26:43.269971 18797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.65:60221
I20260619 10:26:43.270467 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42641
--builtin_ntp_servers=127.9.90.84:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 10:26:43.347901 18988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:43.348105 18988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:43.348132 18988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:43.349221 18988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:43.349257 18988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:43.349289 18988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:43.350561 18988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:43.350788 18988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:43.351073 18988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:43.355676 18994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.355708 18993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.355726 18996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.355731 18988 server_base.cc:1034] running on GCE node
I20260619 10:26:43.356072 18988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:43.356290 18988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:43.357434 18988 hybrid_clock.cc:648] HybridClock initialized: now 1781864803357416 us; error 28 us; skew 500 ppm
I20260619 10:26:43.358578 18988 webserver.cc:460] Webserver started at http://127.9.90.66:43257/ using document root <none> and password file <none>
I20260619 10:26:43.358786 18988 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:43.358832 18988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:43.358949 18988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:43.359935 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "5d8910303bb8437fa18c0e4be4305262"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "4b6047bf8f32f59536d0f2deaa12ae83"
server_key_iv: "b70d728500b81fdd7518132bbda5e629"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.360248 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "5d8910303bb8437fa18c0e4be4305262"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "4b6047bf8f32f59536d0f2deaa12ae83"
server_key_iv: "b70d728500b81fdd7518132bbda5e629"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.361366 18988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:43.362018 19002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.362160 18988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:43.362236 18988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "5d8910303bb8437fa18c0e4be4305262"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "4b6047bf8f32f59536d0f2deaa12ae83"
server_key_iv: "b70d728500b81fdd7518132bbda5e629"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.362291 18988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:43.374611 18988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:43.374874 18988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:43.374990 18988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:43.375186 18988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:43.375476 18988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:43.375507 18988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.375533 18988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:43.375550 18988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.380821 18988 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:40243
I20260619 10:26:43.380860 19115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:40243 every 8 connection(s)
I20260619 10:26:43.381160 18988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:43.384698 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 18988
I20260619 10:26:43.384789 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:43.384954 9577 external_mini_cluster.cc:1428] Setting key 614a6d95a518dfbf1cfad8f4803884a9
I20260619 10:26:43.385910 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42641
--builtin_ntp_servers=127.9.90.84:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 10:26:43.386461 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.381851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:34279 (local address 127.9.90.126:42641)
0619 10:26:43.381920 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:43.381923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:43.382150 (+ 227us) server_negotiation.cc:407] Connection header received
0619 10:26:43.382632 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:43.382633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:43.382643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:43.382706 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:43.383345 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.383826 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.384859 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.385192 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.385269 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:43.386200 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:43.386202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:43.386202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:43.386206 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:43.386223 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:43.386227 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:43.386288 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:43.386349 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 10:26:43.386380 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:26:43.386617 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.381824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42641 (local address 127.9.90.66:34279)
0619 10:26:43.382004 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:43.382043 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:43.382528 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:43.382767 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:43.382771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:43.382779 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:43.383266 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:43.383271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.383889 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.383889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:43.384753 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:43.384757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.385279 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.385281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:43.385429 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:43.386032 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:43.386032 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:43.386072 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:43.386099 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:43.386289 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:43.386292 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:43.386294 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:26:43.386339 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:26:43.386427 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260619 10:26:43.387053 19116 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42641
I20260619 10:26:43.387148 19116 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:43.387342 19116 heartbeater.cc:511] Master 127.9.90.126:42641 requested a full tablet report, sending...
I20260619 10:26:43.387686 18797 ts_manager.cc:194] Registered new tserver with Master: 5d8910303bb8437fa18c0e4be4305262 (127.9.90.66:40243)
I20260619 10:26:43.388137 18797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.66:34279
W20260619 10:26:43.464340 19119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:43.464557 19119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:43.464586 19119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:43.465728 19119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:43.465765 19119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:43.465826 19119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:43.467116 19119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:43.467329 19119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:43.467587 19119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:43.472237 19127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.472267 19125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.472378 19124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.472636 19119 server_base.cc:1034] running on GCE node
I20260619 10:26:43.472811 19119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:43.473023 19119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:43.474162 19119 hybrid_clock.cc:648] HybridClock initialized: now 1781864803474144 us; error 30 us; skew 500 ppm
I20260619 10:26:43.475322 19119 webserver.cc:460] Webserver started at http://127.9.90.67:32893/ using document root <none> and password file <none>
I20260619 10:26:43.475554 19119 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:43.475602 19119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:43.475728 19119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:43.476724 19119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "9eeed5e1d3cc4de1b95cfc718d549514"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "3adf8b235e6ccc790dbce5fab5f066b2"
server_key_iv: "1b88ac5acf0cdf76044055bef3db63d4"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.477033 19119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "9eeed5e1d3cc4de1b95cfc718d549514"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "3adf8b235e6ccc790dbce5fab5f066b2"
server_key_iv: "1b88ac5acf0cdf76044055bef3db63d4"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.478226 19119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:43.478875 19133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.479014 19119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:43.479090 19119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "9eeed5e1d3cc4de1b95cfc718d549514"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "3adf8b235e6ccc790dbce5fab5f066b2"
server_key_iv: "1b88ac5acf0cdf76044055bef3db63d4"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.479144 19119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:43.488145 19119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:43.488374 19119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:43.488487 19119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:43.488668 19119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:43.488935 19119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:43.488966 19119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.488992 19119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:43.489022 19119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.494580 19119 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:33799
I20260619 10:26:43.494633 19246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:33799 every 8 connection(s)
I20260619 10:26:43.494917 19119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:26:43.499744 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.495556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:35319 (local address 127.9.90.126:42641)
0619 10:26:43.495626 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:43.495628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:43.495809 (+ 181us) server_negotiation.cc:407] Connection header received
0619 10:26:43.496327 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:43.496328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:43.496339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:43.496397 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:43.497028 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.497521 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.498538 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.498742 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.498804 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:43.499484 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:43.499485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:43.499486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:43.499490 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:43.499504 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:43.499508 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:43.499546 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:43.499651 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 10:26:43.499680 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:26:43.499843 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.495520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42641 (local address 127.9.90.67:35319)
0619 10:26:43.495697 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:43.495732 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:43.496221 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:43.496467 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:43.496470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:43.496478 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:43.496926 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:43.496931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.497626 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.497627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:43.498449 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:43.498452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:43.498810 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:43.498811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:43.498910 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:43.499367 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:43.499368 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:43.499394 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:43.499416 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:43.499552 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:43.499555 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:43.499557 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:26:43.499603 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:26:43.499686 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260619 10:26:43.500268 19247 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42641
I20260619 10:26:43.500356 19247 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:43.500531 19247 heartbeater.cc:511] Master 127.9.90.126:42641 requested a full tablet report, sending...
I20260619 10:26:43.500909 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 19119
I20260619 10:26:43.500983 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:43.500978 18797 ts_manager.cc:194] Registered new tserver with Master: 9eeed5e1d3cc4de1b95cfc718d549514 (127.9.90.67:33799)
I20260619 10:26:43.501148 9577 external_mini_cluster.cc:1428] Setting key 10f5a1097446e6532796cfd09fda4c98
I20260619 10:26:43.501410 18797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.67:35319
I20260619 10:26:43.502563 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 10:26:43.504385 18847 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:43.503942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.9.90.126:42641)
0619 10:26:43.504015 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:43.504017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:43.504088 (+ 71us) server_negotiation.cc:407] Connection header received
0619 10:26:43.504144 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:43.504145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:43.504155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:43.504199 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:43.504323 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37940: BlockingRecv error: recv got EOF from 127.0.0.1:37940 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 10:26:43.504520 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 18857
I20260619 10:26:43.509038 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 18988
I20260619 10:26:43.512544 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 19119
I20260619 10:26:43.515656 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 18766
2026-06-19T10:26:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (544 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T10:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:43Z Disabled control of system clock
I20260619 10:26:43.539819 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:39051
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:39051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 10:26:43.617251 19258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:43.617446 19258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:43.617475 19258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:43.618854 19258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:43.618899 19258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:43.618911 19258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:43.618921 19258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:43.618932 19258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:43.620301 19258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:39051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:43.620537 19258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:43.620807 19258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:43.625268 19266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.625310 19263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.625442 19258 server_base.cc:1034] running on GCE node
W20260619 10:26:43.625591 19264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.625883 19258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:43.626153 19258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:43.627296 19258 hybrid_clock.cc:648] HybridClock initialized: now 1781864803627283 us; error 27 us; skew 500 ppm
I20260619 10:26:43.628369 19258 webserver.cc:460] Webserver started at http://127.9.90.126:37609/ using document root <none> and password file <none>
I20260619 10:26:43.628552 19258 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:43.628598 19258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:43.628686 19258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:43.629638 19258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "589e715f27254fb9b36c3f97c105b4c9"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "f095a635877fca78c2c95b1df4646600"
server_key_iv: "0d0aa97766b168bf0a28b0083ab1a7ab"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.629987 19258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "589e715f27254fb9b36c3f97c105b4c9"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "f095a635877fca78c2c95b1df4646600"
server_key_iv: "0d0aa97766b168bf0a28b0083ab1a7ab"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.631080 19258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:43.631738 19272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.631892 19258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:43.631969 19258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "589e715f27254fb9b36c3f97c105b4c9"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "f095a635877fca78c2c95b1df4646600"
server_key_iv: "0d0aa97766b168bf0a28b0083ab1a7ab"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.632027 19258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:43.635164 19258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:43.635362 19258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:43.635447 19258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:43.638825 19258 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:39051
I20260619 10:26:43.638857 19324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:39051 every 8 connection(s)
I20260619 10:26:43.639135 19258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:43.639752 19325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:43.642454 19325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9: Bootstrap starting.
I20260619 10:26:43.642987 19325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:43.643241 19325 log.cc:824] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:43.643563 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 19258
I20260619 10:26:43.643656 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:43.643848 9577 external_mini_cluster.cc:1428] Setting key dabf8c1fad55e052e8e37137de4e4c2a
I20260619 10:26:43.643918 19325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9: No bootstrap required, opened a new log
I20260619 10:26:43.646091 19325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589e715f27254fb9b36c3f97c105b4c9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39051 } }
I20260619 10:26:43.646255 19325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:43.646307 19325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589e715f27254fb9b36c3f97c105b4c9, State: Initialized, Role: FOLLOWER
I20260619 10:26:43.646426 19325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589e715f27254fb9b36c3f97c105b4c9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39051 } }
I20260619 10:26:43.646510 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.644359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.9.90.126:39051)
0619 10:26:43.644612 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:43.644620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:43.644648 (+ 28us) server_negotiation.cc:407] Connection header received
0619 10:26:43.645164 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:43.645174 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:43.645224 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:43.645369 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:43.645950 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:43.645954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:43.645955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:43.645959 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:43.646004 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:43.646018 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:43.646097 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:43.646149 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 10:26:43.646202 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260619 10:26:43.646514 19325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:43.646576 19325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:43.646613 19325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:43.647135 19325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589e715f27254fb9b36c3f97c105b4c9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39051 } }
I20260619 10:26:43.647223 19325 leader_election.cc:302] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 589e715f27254fb9b36c3f97c105b4c9; no voters:
I20260619 10:26:43.647368 19325 leader_election.cc:288] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:43.647396 19330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:43.647495 19330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [term 1 LEADER]: Becoming Leader. State: Replica: 589e715f27254fb9b36c3f97c105b4c9, State: Running, Role: LEADER
I20260619 10:26:43.647589 19330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589e715f27254fb9b36c3f97c105b4c9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39051 } }
I20260619 10:26:43.647653 19325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:43.647960 19331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "589e715f27254fb9b36c3f97c105b4c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589e715f27254fb9b36c3f97c105b4c9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39051 } } }
I20260619 10:26:43.647967 19332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 589e715f27254fb9b36c3f97c105b4c9. Latest consensus state: current_term: 1 leader_uuid: "589e715f27254fb9b36c3f97c105b4c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589e715f27254fb9b36c3f97c105b4c9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 39051 } } }
I20260619 10:26:43.648090 19331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:43.648098 19332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:43.648330 19338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:43.648803 19338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:43.650691 19338 catalog_manager.cc:1347] Generated new cluster ID: b22a81fe3996454c9f0506cfc26daabc
I20260619 10:26:43.650743 19338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:26:43.669051 19338 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:43.669243 19338 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:43.673727 19338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 589e715f27254fb9b36c3f97c105b4c9: Generated new TSK 0
I20260619 10:26:43.673920 19338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:26:43.682417 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39051
--builtin_ntp_servers=127.9.90.84:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 10:26:43.761341 19349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:43.761531 19349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:43.761552 19349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:43.762740 19349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:43.762777 19349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:43.762802 19349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:43.764094 19349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:43.764303 19349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:43.764559 19349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:43.769178 19357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.769229 19354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.769266 19349 server_base.cc:1034] running on GCE node
W20260619 10:26:43.769361 19355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.769546 19349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:43.769819 19349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:43.770965 19349 hybrid_clock.cc:648] HybridClock initialized: now 1781864803770952 us; error 29 us; skew 500 ppm
I20260619 10:26:43.772094 19349 webserver.cc:460] Webserver started at http://127.9.90.65:42257/ using document root <none> and password file <none>
I20260619 10:26:43.772300 19349 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:43.772344 19349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:43.772449 19349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:43.773455 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "8ec51226642c46108a8e6389489dea73"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "f9d6ab946efc39319cf4ecfdb477d1ff"
server_key_iv: "e9b537e478e8e7a203206c7a825f9ce6"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.773768 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "8ec51226642c46108a8e6389489dea73"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "f9d6ab946efc39319cf4ecfdb477d1ff"
server_key_iv: "e9b537e478e8e7a203206c7a825f9ce6"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.774927 19349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:43.775574 19363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.775709 19349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:43.775784 19349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "8ec51226642c46108a8e6389489dea73"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "f9d6ab946efc39319cf4ecfdb477d1ff"
server_key_iv: "e9b537e478e8e7a203206c7a825f9ce6"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.775846 19349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:43.778982 19349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:43.779211 19349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:43.779323 19349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:43.779501 19349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:43.779769 19349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:43.779799 19349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.779824 19349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:43.779848 19349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.784907 19349 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:41061
I20260619 10:26:43.784965 19476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:41061 every 8 connection(s)
I20260619 10:26:43.785230 19349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:26:43.786662 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 19349
I20260619 10:26:43.786760 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:43.786959 9577 external_mini_cluster.cc:1428] Setting key d3fc81be44d6131bb6dec6d79e5dfbd5
I20260619 10:26:43.787751 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39051
--builtin_ntp_servers=127.9.90.84:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 10:26:43.788051 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.785869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:48697 (local address 127.9.90.126:39051)
0619 10:26:43.785918 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:43.785920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:43.786141 (+ 221us) server_negotiation.cc:407] Connection header received
0619 10:26:43.786591 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:43.786592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:43.786597 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:43.786639 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:43.787795 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:43.787797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:43.787797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:43.787801 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:43.787817 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:43.787821 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:43.787866 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:43.787969 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 10:26:43.787985 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:43.788187 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.785852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:39051 (local address 127.9.90.65:48697)
0619 10:26:43.786027 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:43.786059 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:43.786503 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:43.786728 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:43.786733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:43.786745 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:43.787660 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:43.787661 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:43.787687 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:43.787713 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:43.787890 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:43.787895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:43.787898 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:43.787967 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:26:43.788017 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260619 10:26:43.788594 19477 heartbeater.cc:347] Connected to a master server at 127.9.90.126:39051
I20260619 10:26:43.788678 19477 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:43.788796 19477 heartbeater.cc:511] Master 127.9.90.126:39051 requested a full tablet report, sending...
I20260619 10:26:43.789266 19289 ts_manager.cc:194] Registered new tserver with Master: 8ec51226642c46108a8e6389489dea73 (127.9.90.65:41061)
W20260619 10:26:43.864658 19480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:43.864841 19480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:43.864861 19480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:43.865967 19480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:43.866003 19480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:43.866026 19480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:43.867345 19480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:43.867571 19480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:43.867870 19480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:43.872498 19486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.872536 19485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.872614 19480 server_base.cc:1034] running on GCE node
W20260619 10:26:43.872541 19488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.872900 19480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:43.873081 19480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:43.874205 19480 hybrid_clock.cc:648] HybridClock initialized: now 1781864803874194 us; error 29 us; skew 500 ppm
I20260619 10:26:43.875329 19480 webserver.cc:460] Webserver started at http://127.9.90.66:42157/ using document root <none> and password file <none>
I20260619 10:26:43.875519 19480 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:43.875560 19480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:43.875686 19480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:43.876680 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "aa887020aa8c4506a8627164fbbb519b"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "4e3086746f139b2d0f7cd147b131eee9"
server_key_iv: "bc3529ad0adfe10f529e7dbf99762aca"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.876988 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "aa887020aa8c4506a8627164fbbb519b"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "4e3086746f139b2d0f7cd147b131eee9"
server_key_iv: "bc3529ad0adfe10f529e7dbf99762aca"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.878237 19480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:26:43.878912 19494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.879052 19480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:43.879153 19480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "aa887020aa8c4506a8627164fbbb519b"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "4e3086746f139b2d0f7cd147b131eee9"
server_key_iv: "bc3529ad0adfe10f529e7dbf99762aca"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.879215 19480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:43.882298 19480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:43.882513 19480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:43.882635 19480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:43.882809 19480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:43.883064 19480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:43.883095 19480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.883118 19480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:43.883136 19480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.888417 19480 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:46271
I20260619 10:26:43.888466 19607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:46271 every 8 connection(s)
I20260619 10:26:43.888749 19480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:26:43.891500 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.889405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:54873 (local address 127.9.90.126:39051)
0619 10:26:43.889499 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:43.889501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:43.889664 (+ 163us) server_negotiation.cc:407] Connection header received
0619 10:26:43.890148 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:43.890150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:43.890155 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:43.890206 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:43.891211 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:43.891212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:43.891213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:43.891216 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:43.891252 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:43.891256 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:43.891316 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:43.891395 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 10:26:43.891409 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 10:26:43.891673 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.889384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:39051 (local address 127.9.90.66:54873)
0619 10:26:43.889540 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:43.889571 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:43.890060 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:43.890258 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:43.890261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:43.890269 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:43.891069 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:43.891069 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:43.891095 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:43.891122 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:43.891320 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:43.891324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:43.891327 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:43.891395 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 10:26:43.891455 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260619 10:26:43.892194 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 19480
I20260619 10:26:43.892343 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:43.892603 9577 external_mini_cluster.cc:1428] Setting key 641aac5e4539b1072556fb6d9b1bc4c3
I20260619 10:26:43.892786 19608 heartbeater.cc:347] Connected to a master server at 127.9.90.126:39051
I20260619 10:26:43.892877 19608 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:43.893005 19608 heartbeater.cc:511] Master 127.9.90.126:39051 requested a full tablet report, sending...
I20260619 10:26:43.893379 19288 ts_manager.cc:194] Registered new tserver with Master: aa887020aa8c4506a8627164fbbb519b (127.9.90.66:46271)
I20260619 10:26:43.894500 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39051
--builtin_ntp_servers=127.9.90.84:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 10:26:43.972556 19611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:43.972739 19611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:43.972759 19611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:43.973928 19611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:43.973966 19611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:43.973992 19611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:43.975297 19611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:39051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:43.975509 19611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:43.975764 19611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:43.980394 19617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.980409 19616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:43.980499 19619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:43.980412 19611 server_base.cc:1034] running on GCE node
I20260619 10:26:43.980778 19611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:43.980976 19611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:43.982103 19611 hybrid_clock.cc:648] HybridClock initialized: now 1781864803982086 us; error 37 us; skew 500 ppm
I20260619 10:26:43.983240 19611 webserver.cc:460] Webserver started at http://127.9.90.67:37571/ using document root <none> and password file <none>
I20260619 10:26:43.983445 19611 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:43.983490 19611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:43.983611 19611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:43.984570 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "47491ff8fcf34a738946abf04f8daeeb"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "d5d8e8f6c7b6e30235583603eefcbe26"
server_key_iv: "362dd86a3c136fd240bcc7518c409c87"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.984871 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "47491ff8fcf34a738946abf04f8daeeb"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "d5d8e8f6c7b6e30235583603eefcbe26"
server_key_iv: "362dd86a3c136fd240bcc7518c409c87"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.986054 19611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:26:43.986711 19625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.986856 19611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:43.986931 19611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "47491ff8fcf34a738946abf04f8daeeb"
format_stamp: "Formatted at 2026-06-19 10:26:43 on dist-test-slave-hw7q"
server_key: "d5d8e8f6c7b6e30235583603eefcbe26"
server_key_iv: "362dd86a3c136fd240bcc7518c409c87"
server_key_version: "encryptionkey@0"
I20260619 10:26:43.986987 19611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:43.990146 19611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:43.990378 19611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:43.990489 19611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:43.990670 19611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:43.990929 19611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:43.990960 19611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.990985 19611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:43.991008 19611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:43.996248 19611 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:43965
I20260619 10:26:43.996296 19738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:43965 every 8 connection(s)
I20260619 10:26:43.996582 19611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:26:43.998411 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 19611
I20260619 10:26:43.998503 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:43.998700 9577 external_mini_cluster.cc:1428] Setting key fff2c2dced9cc9281f721c29c4d6940c
I20260619 10:26:43.999505 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.997191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:34879 (local address 127.9.90.126:39051)
0619 10:26:43.997234 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:43.997236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:43.997482 (+ 246us) server_negotiation.cc:407] Connection header received
0619 10:26:43.998015 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:43.998017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:43.998024 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:43.998082 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:43.999274 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:43.999275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:43.999275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:26:43.999279 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:43.999294 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:26:43.999298 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:43.999352 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:43.999422 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:26:43.999438 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:26:43.999706 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:43.997200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:39051 (local address 127.9.90.67:34879)
0619 10:26:43.997362 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:43.997394 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:43.997919 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:43.998153 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:43.998157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:43.998166 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:43.999143 (+ 977us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:26:43.999143 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:43.999171 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:26:43.999197 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:43.999355 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:43.999359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:43.999362 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:43.999429 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 10:26:43.999488 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260619 10:26:44.000080 19739 heartbeater.cc:347] Connected to a master server at 127.9.90.126:39051
I20260619 10:26:44.000165 19739 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:44.000289 19739 heartbeater.cc:511] Master 127.9.90.126:39051 requested a full tablet report, sending...
I20260619 10:26:44.000651 19289 ts_manager.cc:194] Registered new tserver with Master: 47491ff8fcf34a738946abf04f8daeeb (127.9.90.67:43965)
I20260619 10:26:44.010442 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 10:26:44.012272 19329 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:26:44.011768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.9.90.126:39051)
0619 10:26:44.011825 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:44.011827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:44.011932 (+ 105us) server_negotiation.cc:407] Connection header received
0619 10:26:44.011996 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:44.011997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:44.012001 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:44.012037 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:44.012203 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44622: BlockingRecv error: recv got EOF from 127.0.0.1:44622 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:26:44.012333 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 19349
I20260619 10:26:44.016366 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 19480
I20260619 10:26:44.021629 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 19611
I20260619 10:26:44.025060 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 19258
2026-06-19T10:26:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (509 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:44 dist-test-slave-hw7q krb5kdc[19749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:44 dist-test-slave-hw7q krb5kdc[19749](info): set up 2 sockets
Jun 19 10:26:44 dist-test-slave-hw7q krb5kdc[19749](info): commencing operation
krb5kdc: starting...
W20260619 10:26:44.986337 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[19749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:26:45.095800 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33417
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:33417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:45.172035 19762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:45.172235 19762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:45.172256 19762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:45.173388 19762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:26:45.173426 19762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:45.173439 19762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:45.173450 19762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:26:45.173462 19762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:26:45.174937 19762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:33417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:45.175153 19762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:45.175355 19762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:45.179699 19770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:45.179721 19767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:45.179762 19762 server_base.cc:1034] running on GCE node
W20260619 10:26:45.179700 19768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:45.180068 19762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:45.180337 19762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:45.181475 19762 hybrid_clock.cc:648] HybridClock initialized: now 1781864805181463 us; error 27 us; skew 500 ppm
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[19749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864805, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:45.183702 19762 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:26:45.184116 19762 webserver.cc:460] Webserver started at http://127.9.90.126:43299/ using document root <none> and password file <none>
I20260619 10:26:45.184299 19762 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:45.184345 19762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:45.184463 19762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:45.185422 19762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "a57eedcff9574dbd93ea0fc81815ac35"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "cf9ef58b9721aeb89b5240255535c4c1"
server_key_iv: "531ea067a3206193d0bfa298434a491d"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.185729 19762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "a57eedcff9574dbd93ea0fc81815ac35"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "cf9ef58b9721aeb89b5240255535c4c1"
server_key_iv: "531ea067a3206193d0bfa298434a491d"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.186913 19762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:26:45.187530 19777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.187673 19762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:45.187748 19762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "a57eedcff9574dbd93ea0fc81815ac35"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "cf9ef58b9721aeb89b5240255535c4c1"
server_key_iv: "531ea067a3206193d0bfa298434a491d"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.187805 19762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:45.195999 19762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:45.197705 19762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:45.197845 19762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:45.201268 19762 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:33417
I20260619 10:26:45.201299 19829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:33417 every 8 connection(s)
I20260619 10:26:45.201573 19762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:26:45.202577 19830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:45.204635 19830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35: Bootstrap starting.
I20260619 10:26:45.205150 19830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:45.205389 19830 log.cc:824] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:45.206058 19830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35: No bootstrap required, opened a new log
I20260619 10:26:45.207543 19830 raft_consensus.cc:348] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57eedcff9574dbd93ea0fc81815ac35" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33417 } }
I20260619 10:26:45.207666 19830 raft_consensus.cc:374] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:45.207687 19830 raft_consensus.cc:729] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57eedcff9574dbd93ea0fc81815ac35, State: Initialized, Role: FOLLOWER
I20260619 10:26:45.207818 19830 consensus_queue.cc:260] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57eedcff9574dbd93ea0fc81815ac35" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33417 } }
I20260619 10:26:45.207880 19830 raft_consensus.cc:388] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:26:45.207911 19830 raft_consensus.cc:482] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:26:45.207954 19830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:45.208670 19830 raft_consensus.cc:504] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57eedcff9574dbd93ea0fc81815ac35" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33417 } }
I20260619 10:26:45.208750 19830 leader_election.cc:302] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a57eedcff9574dbd93ea0fc81815ac35; no voters:
I20260619 10:26:45.208884 19830 leader_election.cc:288] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:26:45.208920 19833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:45.209067 19833 raft_consensus.cc:686] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [term 1 LEADER]: Becoming Leader. State: Replica: a57eedcff9574dbd93ea0fc81815ac35, State: Running, Role: LEADER
I20260619 10:26:45.209121 19830 sys_catalog.cc:564] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:26:45.209152 19833 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57eedcff9574dbd93ea0fc81815ac35" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33417 } }
I20260619 10:26:45.209455 19834 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a57eedcff9574dbd93ea0fc81815ac35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57eedcff9574dbd93ea0fc81815ac35" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33417 } } }
I20260619 10:26:45.209475 19835 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57eedcff9574dbd93ea0fc81815ac35. Latest consensus state: current_term: 1 leader_uuid: "a57eedcff9574dbd93ea0fc81815ac35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57eedcff9574dbd93ea0fc81815ac35" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33417 } } }
I20260619 10:26:45.209558 19835 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:45.209698 19834 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35 [sys.catalog]: This master's current role is: LEADER
I20260619 10:26:45.209710 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 19762
I20260619 10:26:45.209822 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:26:45.209820 19838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:26:45.209996 9577 external_mini_cluster.cc:1428] Setting key e5b4dfa1bd0b8492b1786a0f7f1feeeb
I20260619 10:26:45.210295 19838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:26:45.211733 19838 catalog_manager.cc:1347] Generated new cluster ID: 73607cc5f8234ef6a7c395cee04710f8
I20260619 10:26:45.211781 19838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[19749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:45.221292 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.210617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.9.90.126:33417)
0619 10:26:45.211249 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.211256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.211278 (+ 22us) server_negotiation.cc:407] Connection header received
0619 10:26:45.211379 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.211386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.211439 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.211571 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:45.212280 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.213614 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.214687 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.215241 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.215328 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.217955 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:45.217959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:45.217959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:45.217970 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:45.219817 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.220157 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.220160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.220160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.220263 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.220427 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.220428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.220429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.220713 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:45.220862 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.220934 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.221070 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":63,"threads_started":1}
I20260619 10:26:45.222620 19838 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:26:45.223305 19838 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:26:45.226388 19838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35: Generated new TSK 0
I20260619 10:26:45.226526 19838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:26:45.252936 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33417
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:45.331905 19856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:45.332105 19856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:45.332129 19856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:45.333237 19856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:45.333278 19856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:45.333305 19856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:26:45.334710 19856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:45.334906 19856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:45.335083 19856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:45.339694 19861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:45.339754 19862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:45.339859 19856 server_base.cc:1034] running on GCE node
W20260619 10:26:45.339731 19864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:45.340086 19856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:45.340296 19856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:45.341429 19856 hybrid_clock.cc:648] HybridClock initialized: now 1781864805341417 us; error 23 us; skew 500 ppm
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[19749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864805, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:45.343741 19856 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:26:45.344166 19856 webserver.cc:460] Webserver started at http://127.9.90.65:33127/ using document root <none> and password file <none>
I20260619 10:26:45.344350 19856 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:45.344394 19856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:45.344512 19856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:45.345465 19856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "d1d4ea69aeb84d58af03128ab5286f59"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "a83a0592b278e8b58862ebbd4c90e409"
server_key_iv: "5923fa975b346baf8c4ced51aab2e039"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.345770 19856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "d1d4ea69aeb84d58af03128ab5286f59"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "a83a0592b278e8b58862ebbd4c90e409"
server_key_iv: "5923fa975b346baf8c4ced51aab2e039"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.346895 19856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:26:45.347573 19871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.347764 19856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:45.347848 19856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "d1d4ea69aeb84d58af03128ab5286f59"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "a83a0592b278e8b58862ebbd4c90e409"
server_key_iv: "5923fa975b346baf8c4ced51aab2e039"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.347908 19856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:45.370365 19856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:45.372046 19856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:45.372176 19856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:45.372386 19856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:45.372690 19856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:45.372725 19856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.372752 19856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:45.372769 19856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.378333 19856 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:37735
I20260619 10:26:45.378374 19984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:37735 every 8 connection(s)
I20260619 10:26:45.378664 19856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[19749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864805, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:45.386843 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.379318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:45829 (local address 127.9.90.126:33417)
0619 10:26:45.379370 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.379372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.379550 (+ 178us) server_negotiation.cc:407] Connection header received
0619 10:26:45.380010 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.380011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.380022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.380072 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:45.380911 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.381399 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.382377 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.382565 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.382617 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.384403 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:45.384404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:45.384405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:45.384412 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:45.385627 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.386041 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.386042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.386043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.386088 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.386323 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.386323 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.386324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.386411 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:45.386454 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.386718 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.386778 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:26:45.387079 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.379295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33417 (local address 127.9.90.65:45829)
0619 10:26:45.379435 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:45.379474 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:45.379940 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:45.380127 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:45.380130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:45.380660 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:45.380842 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.380846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.381479 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.381480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.382322 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.382324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.382625 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.382626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.382721 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.383097 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:45.383105 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:45.384318 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:45.385706 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:45.385717 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:45.385719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:45.385974 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:45.386162 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:45.386163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:45.386163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:45.386270 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:45.386482 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:45.386487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:45.386602 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 10:26:45.386691 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 10:26:45.386806 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260619 10:26:45.387188 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 19856
I20260619 10:26:45.387272 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:26:45.387454 9577 external_mini_cluster.cc:1428] Setting key 82102fb89852c29fa248c19766bace23
I20260619 10:26:45.387552 19985 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33417
I20260619 10:26:45.387642 19985 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:45.387813 19985 heartbeater.cc:511] Master 127.9.90.126:33417 requested a full tablet report, sending...
I20260619 10:26:45.388314 19794 ts_manager.cc:194] Registered new tserver with Master: d1d4ea69aeb84d58af03128ab5286f59 (127.9.90.65:37735)
I20260619 10:26:45.389081 19794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:45829
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:26:45.408511 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33417
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:45.486052 19990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:45.486260 19990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:45.486289 19990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:45.487448 19990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:45.487488 19990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:45.487517 19990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:26:45.488903 19990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:45.489130 19990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:45.489326 19990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:26:45.494030 19990 server_base.cc:1034] running on GCE node
W20260619 10:26:45.494056 19995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:45.494140 19998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:45.493999 19996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:45.494400 19990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:45.494639 19990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:45.495836 19990 hybrid_clock.cc:648] HybridClock initialized: now 1781864805495814 us; error 34 us; skew 500 ppm
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[19749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864805, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:45.498378 19990 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:26:45.498796 19990 webserver.cc:460] Webserver started at http://127.9.90.66:42347/ using document root <none> and password file <none>
I20260619 10:26:45.499003 19990 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:45.499053 19990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:45.499179 19990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:45.500252 19990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "bd5789d2f60b4e858c9127a2e0482b10"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "f8f5ddf166df8ff928bdc2c12ce4b3fd"
server_key_iv: "b8aa646e04c134f87adbdf075b4c498a"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.500572 19990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "bd5789d2f60b4e858c9127a2e0482b10"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "f8f5ddf166df8ff928bdc2c12ce4b3fd"
server_key_iv: "b8aa646e04c134f87adbdf075b4c498a"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.501837 19990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:26:45.502523 20005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.502665 19990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:26:45.502751 19990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "bd5789d2f60b4e858c9127a2e0482b10"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "f8f5ddf166df8ff928bdc2c12ce4b3fd"
server_key_iv: "b8aa646e04c134f87adbdf075b4c498a"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.502815 19990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:45.515647 19990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:45.517205 19990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:45.517325 19990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:45.517515 19990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:45.517776 19990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:45.517863 19990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.517894 19990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:45.517917 19990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.523234 19990 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:35789
I20260619 10:26:45.523288 20118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:35789 every 8 connection(s)
I20260619 10:26:45.523552 19990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[19749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864805, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:45.531525 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.524136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:47123 (local address 127.9.90.126:33417)
0619 10:26:45.524185 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.524187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.524386 (+ 199us) server_negotiation.cc:407] Connection header received
0619 10:26:45.524820 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.524822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.524831 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.524877 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:45.525703 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.526195 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.527139 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.527321 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.527366 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.529231 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:45.529233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:45.529233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:45.529241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:45.530389 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.530719 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.530719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.530720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.530768 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.530988 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.530989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.530990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.531110 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:45.531173 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.531394 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.531462 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:26:45.531607 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.524141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33417 (local address 127.9.90.66:47123)
0619 10:26:45.524279 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:45.524311 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:45.524750 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:45.524936 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:45.524939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:45.525458 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:45.525637 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.525641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.526261 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.526262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.527078 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.527080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.527375 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.527376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.527465 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.527869 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:45.527877 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:45.529150 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:45.530461 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:45.530467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:45.530468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:45.530661 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:45.530812 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:45.530812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:45.530813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:45.530893 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:45.531168 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:45.531171 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:45.531274 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 10:26:45.531348 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:26:45.531454 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260619 10:26:45.531962 20119 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33417
I20260619 10:26:45.532044 20119 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:45.532176 20119 heartbeater.cc:511] Master 127.9.90.126:33417 requested a full tablet report, sending...
I20260619 10:26:45.532503 19794 ts_manager.cc:194] Registered new tserver with Master: bd5789d2f60b4e858c9127a2e0482b10 (127.9.90.66:35789)
I20260619 10:26:45.532613 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 19990
I20260619 10:26:45.532703 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:26:45.532876 9577 external_mini_cluster.cc:1428] Setting key d2dff7db4cf5a5d30297e8eb06ce99d7
I20260619 10:26:45.532903 19794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:47123
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:26:45.553712 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33417
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:26:45.630949 20124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:26:45.631140 20124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:26:45.631161 20124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:26:45.632261 20124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:26:45.632300 20124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:26:45.632324 20124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:26:45.633708 20124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:26:45.633929 20124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:26:45.634132 20124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:26:45.638862 20130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:45.638914 20132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:26:45.638892 20129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:26:45.638885 20124 server_base.cc:1034] running on GCE node
I20260619 10:26:45.639241 20124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:26:45.639418 20124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:26:45.640542 20124 hybrid_clock.cc:648] HybridClock initialized: now 1781864805640531 us; error 27 us; skew 500 ppm
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[19749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864805, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:26:45.642892 20124 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:26:45.643329 20124 webserver.cc:460] Webserver started at http://127.9.90.67:40661/ using document root <none> and password file <none>
I20260619 10:26:45.643532 20124 fs_manager.cc:359] Metadata directory not provided
I20260619 10:26:45.643579 20124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:26:45.643697 20124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:26:45.644658 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "36608394994d4e3fb198f11c26360acb"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "c569dcba69a055e15cdea1f983ab2879"
server_key_iv: "ab137945057f0937d70f8cea2751673d"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.644968 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "36608394994d4e3fb198f11c26360acb"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "c569dcba69a055e15cdea1f983ab2879"
server_key_iv: "ab137945057f0937d70f8cea2751673d"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.646188 20124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:26:45.646862 20139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.647004 20124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:26:45.647074 20124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "36608394994d4e3fb198f11c26360acb"
format_stamp: "Formatted at 2026-06-19 10:26:45 on dist-test-slave-hw7q"
server_key: "c569dcba69a055e15cdea1f983ab2879"
server_key_iv: "ab137945057f0937d70f8cea2751673d"
server_key_version: "encryptionkey@0"
I20260619 10:26:45.647148 20124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:26:45.668510 20124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:26:45.670569 20124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:26:45.670702 20124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:26:45.670907 20124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:26:45.671187 20124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:26:45.671221 20124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.671248 20124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:26:45.671269 20124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:26:45.676659 20124 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:35431
I20260619 10:26:45.676700 20252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:35431 every 8 connection(s)
I20260619 10:26:45.676985 20124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:26:45.677847 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 20124
I20260619 10:26:45.677954 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:26:45.678135 9577 external_mini_cluster.cc:1428] Setting key ef43f690438a7fcb76f48bd3a9810253
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[19749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864805, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:26:45.685775 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.677617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:58717 (local address 127.9.90.126:33417)
0619 10:26:45.677708 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.677710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.677905 (+ 195us) server_negotiation.cc:407] Connection header received
0619 10:26:45.678561 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.678562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.678575 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.678636 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:45.679582 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.680108 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.681017 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.681201 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.681256 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.683275 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:45.683277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:45.683277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:45.683285 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:45.684577 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.684941 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.684942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.684943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.685001 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.685215 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.685216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.685216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.685338 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:45.685397 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.685639 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.685710 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 10:26:45.685904 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.677588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33417 (local address 127.9.90.67:58717)
0619 10:26:45.677748 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:45.677819 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:45.678449 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:45.678714 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:45.678718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:45.679313 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:26:45.679498 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.679503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.680173 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.680174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.680937 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.680940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.681262 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.681262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.681352 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.681834 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:26:45.681843 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:26:45.683188 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:26:45.684656 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:45.684661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:45.684663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:45.684856 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:45.685053 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:26:45.685054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:26:45.685054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:26:45.685145 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:26:45.685391 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:26:45.685394 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:26:45.685502 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 10:26:45.685576 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:26:45.685691 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260619 10:26:45.686357 20253 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33417
I20260619 10:26:45.686466 20253 heartbeater.cc:464] Registering TS with master...
I20260619 10:26:45.686622 20253 heartbeater.cc:511] Master 127.9.90.126:33417 requested a full tablet report, sending...
I20260619 10:26:45.686972 19794 ts_manager.cc:194] Registered new tserver with Master: 36608394994d4e3fb198f11c26360acb (127.9.90.67:35431)
I20260619 10:26:45.687321 19794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:58717
I20260619 10:26:45.689615 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:26:45.696174 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.690936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.9.90.126:33417)
0619 10:26:45.690998 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.691000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.691042 (+ 42us) server_negotiation.cc:407] Connection header received
0619 10:26:45.691090 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.691090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.691098 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.691141 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:26:45.691625 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.692121 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.692798 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.693017 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.693072 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.694210 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:26:45.694211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:26:45.694212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:26:45.694218 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:26:45.695269 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.695567 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.695568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.695569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.695626 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:26:45.695768 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:26:45.695769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:26:45.695769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:26:45.695897 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:26:45.695968 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.696046 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.696111 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:26:45.697674 19794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:26:45.699205 19794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:26:45.704890 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.701012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:35431 (local address 127.0.0.1:60326)
0619 10:26:45.701304 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:45.701313 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:45.701350 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:45.701633 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:45.701645 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:45.701647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:45.701688 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.701689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.702968 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.702969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.703739 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.703741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.704651 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.704652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.704716 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.704727 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 10:26:45.704781 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 10:26:45.704817 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260619 10:26:45.704937 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.701117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60326 (local address 127.9.90.67:35431)
0619 10:26:45.701502 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.701506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.701514 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:45.701547 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.701549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.701562 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.701653 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:45.701740 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.702873 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.703830 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.704564 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.704645 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.704702 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.704813 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.704853 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":74,"threads_started":1}
I20260619 10:26:45.705338 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.700991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:37735 (local address 127.0.0.1:55668)
0619 10:26:45.701095 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:45.701132 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:45.701187 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:45.701463 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:45.701466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:45.701468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:45.701530 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.701532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.702927 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.702928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.704018 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.704020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.704876 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.704877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.704929 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.704932 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:26:45.705231 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 10:26:45.705274 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":26,"thread_start_us":55,"threads_started":1}
I20260619 10:26:45.705509 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.701174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.9.90.65:37735)
0619 10:26:45.701309 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.701314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.701325 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:26:45.701366 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.701368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.701384 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.701458 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:45.701598 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.702822 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.704110 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.704793 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.704864 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.704918 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.705291 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.705331 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260619 10:26:45.705690 20187 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc618f99fe24bd4b7058a6ef01caea6 (DEFAULT_TABLE table=test-table [id=4342bec7b9fe44b7afaf7c5fd8f08d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:45.706034 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.701194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:35789 (local address 127.0.0.1:40420)
0619 10:26:45.701332 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:45.701342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:45.701390 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:45.701822 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:45.701823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:45.701825 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:45.701854 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.701855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.702754 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.702755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.704019 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.704020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.705764 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.705765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.705840 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.705844 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:45.705931 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 10:26:45.705974 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":5,"thread_start_us":77,"threads_started":1}
I20260619 10:26:45.706049 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.701412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.9.90.66:35789)
0619 10:26:45.701586 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.701591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.701599 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:45.701647 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.701649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.701663 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.701735 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:45.701924 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.702662 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.704955 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.705672 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.705738 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.705840 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.705916 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.705956 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20260619 10:26:45.706091 20187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc618f99fe24bd4b7058a6ef01caea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:45.706804 20053 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc618f99fe24bd4b7058a6ef01caea6 (DEFAULT_TABLE table=test-table [id=4342bec7b9fe44b7afaf7c5fd8f08d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:45.707067 20053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc618f99fe24bd4b7058a6ef01caea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:45.708578 20267 tablet_bootstrap.cc:492] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: Bootstrap starting.
I20260619 10:26:45.709177 19919 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc618f99fe24bd4b7058a6ef01caea6 (DEFAULT_TABLE table=test-table [id=4342bec7b9fe44b7afaf7c5fd8f08d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:26:45.709422 20267 tablet_bootstrap.cc:654] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:45.709486 19919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc618f99fe24bd4b7058a6ef01caea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:26:45.709519 20268 tablet_bootstrap.cc:492] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: Bootstrap starting.
I20260619 10:26:45.710191 20267 log.cc:824] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:45.710322 20268 tablet_bootstrap.cc:654] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:45.710734 20268 log.cc:824] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:45.711400 20268 tablet_bootstrap.cc:492] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: No bootstrap required, opened a new log
I20260619 10:26:45.711458 20268 ts_tablet_manager.cc:1397] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:45.712194 20271 tablet_bootstrap.cc:492] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59: Bootstrap starting.
I20260619 10:26:45.712635 20267 tablet_bootstrap.cc:492] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: No bootstrap required, opened a new log
I20260619 10:26:45.712695 20267 ts_tablet_manager.cc:1397] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 10:26:45.712846 20271 tablet_bootstrap.cc:654] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59: Neither blocks nor log segments found. Creating new log.
I20260619 10:26:45.713343 20271 log.cc:824] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59: Log is configured to *not* fsync() on all Append() calls
I20260619 10:26:45.713351 20268 raft_consensus.cc:348] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } }
I20260619 10:26:45.713501 20268 raft_consensus.cc:374] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:45.713537 20268 raft_consensus.cc:729] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5789d2f60b4e858c9127a2e0482b10, State: Initialized, Role: FOLLOWER
I20260619 10:26:45.713649 20268 consensus_queue.cc:260] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } }
I20260619 10:26:45.713912 20268 ts_tablet_manager.cc:1428] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 10:26:45.713982 20119 heartbeater.cc:503] Master 127.9.90.126:33417 was elected leader, sending a full tablet report...
I20260619 10:26:45.714318 20267 raft_consensus.cc:348] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } }
I20260619 10:26:45.714448 20271 tablet_bootstrap.cc:492] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59: No bootstrap required, opened a new log
I20260619 10:26:45.714468 20267 raft_consensus.cc:374] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:45.714496 20271 ts_tablet_manager.cc:1397] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:26:45.714499 20267 raft_consensus.cc:729] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36608394994d4e3fb198f11c26360acb, State: Initialized, Role: FOLLOWER
I20260619 10:26:45.714608 20267 consensus_queue.cc:260] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } }
I20260619 10:26:45.714865 20267 ts_tablet_manager.cc:1428] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:26:45.714972 20253 heartbeater.cc:503] Master 127.9.90.126:33417 was elected leader, sending a full tablet report...
I20260619 10:26:45.716014 20271 raft_consensus.cc:348] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } }
I20260619 10:26:45.716128 20271 raft_consensus.cc:374] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:26:45.716147 20271 raft_consensus.cc:729] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d4ea69aeb84d58af03128ab5286f59, State: Initialized, Role: FOLLOWER
I20260619 10:26:45.716246 20271 consensus_queue.cc:260] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } }
I20260619 10:26:45.716459 20271 ts_tablet_manager.cc:1428] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:26:45.716481 19985 heartbeater.cc:503] Master 127.9.90.126:33417 was elected leader, sending a full tablet report...
W20260619 10:26:45.774088 20120 tablet.cc:2369] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:26:45.776039 20273 raft_consensus.cc:482] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:26:45.776177 20273 raft_consensus.cc:504] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } }
I20260619 10:26:45.776661 20273 leader_election.cc:288] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1d4ea69aeb84d58af03128ab5286f59 (127.9.90.65:37735), 36608394994d4e3fb198f11c26360acb (127.9.90.67:35431)
I20260619 10:26:45.779284 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.776828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:37735 (local address 127.9.90.66:45895)
0619 10:26:45.776875 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:45.776886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:45.776926 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:45.777093 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:45.777094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:45.777096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:45.777176 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.777178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.777773 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.777774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.778525 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.778527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.779044 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.779045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.779121 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.779125 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:26:45.779170 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:26:45.779210 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 10:26:45.779284 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.776920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:45895 (local address 127.9.90.65:37735)
0619 10:26:45.776965 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.776966 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.776977 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:26:45.777005 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.777006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.777018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.777077 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:45.777253 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.777708 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.778612 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.778980 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.779040 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.779050 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.779186 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.779220 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:45.779568 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.776948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:40983 (local address 127.9.90.67:35431)
0619 10:26:45.776988 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.776989 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.777229 (+ 240us) server_negotiation.cc:407] Connection header received
0619 10:26:45.777259 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.777260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.777271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.777321 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:26:45.777448 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.777942 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.778805 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.779181 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.779238 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.779248 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.779450 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.779491 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:26:45.779647 19939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc618f99fe24bd4b7058a6ef01caea6" candidate_uuid: "bd5789d2f60b4e858c9127a2e0482b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d4ea69aeb84d58af03128ab5286f59" is_pre_election: true
I20260619 10:26:45.779768 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.776829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:35431 (local address 127.9.90.66:40983)
0619 10:26:45.777162 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:45.777170 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:26:45.777214 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:26:45.777329 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:26:45.777329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:26:45.777330 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:26:45.777394 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.777395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.778010 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.778011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.778722 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:26:45.778724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.779252 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.779253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:26:45.779321 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.779323 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:26:45.779657 (+ 334us) client_negotiation.cc:240] Negotiation successful
0619 10:26:45.779701 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":162,"thread_start_us":81,"threads_started":1}
I20260619 10:26:45.779819 19939 raft_consensus.cc:2454] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5789d2f60b4e858c9127a2e0482b10 in term 0.
I20260619 10:26:45.780061 20007 leader_election.cc:302] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd5789d2f60b4e858c9127a2e0482b10, d1d4ea69aeb84d58af03128ab5286f59; no voters:
I20260619 10:26:45.780072 20207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc618f99fe24bd4b7058a6ef01caea6" candidate_uuid: "bd5789d2f60b4e858c9127a2e0482b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36608394994d4e3fb198f11c26360acb" is_pre_election: true
I20260619 10:26:45.780207 20207 raft_consensus.cc:2454] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5789d2f60b4e858c9127a2e0482b10 in term 0.
I20260619 10:26:45.780223 20273 raft_consensus.cc:2781] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:26:45.780264 20273 raft_consensus.cc:482] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:26:45.780292 20273 raft_consensus.cc:3037] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:45.780987 20273 raft_consensus.cc:504] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } }
I20260619 10:26:45.781095 20273 leader_election.cc:288] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [CANDIDATE]: Term 1 election: Requested vote from peers d1d4ea69aeb84d58af03128ab5286f59 (127.9.90.65:37735), 36608394994d4e3fb198f11c26360acb (127.9.90.67:35431)
I20260619 10:26:45.781250 19939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc618f99fe24bd4b7058a6ef01caea6" candidate_uuid: "bd5789d2f60b4e858c9127a2e0482b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d4ea69aeb84d58af03128ab5286f59"
I20260619 10:26:45.781284 20207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc618f99fe24bd4b7058a6ef01caea6" candidate_uuid: "bd5789d2f60b4e858c9127a2e0482b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36608394994d4e3fb198f11c26360acb"
I20260619 10:26:45.781329 19939 raft_consensus.cc:3037] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:45.781363 20207 raft_consensus.cc:3037] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:26:45.782042 19939 raft_consensus.cc:2454] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5789d2f60b4e858c9127a2e0482b10 in term 1.
I20260619 10:26:45.782136 20207 raft_consensus.cc:2454] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5789d2f60b4e858c9127a2e0482b10 in term 1.
I20260619 10:26:45.782200 20007 leader_election.cc:302] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd5789d2f60b4e858c9127a2e0482b10, d1d4ea69aeb84d58af03128ab5286f59; no voters:
I20260619 10:26:45.782280 20273 raft_consensus.cc:2781] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:26:45.782419 20273 raft_consensus.cc:686] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 1 LEADER]: Becoming Leader. State: Replica: bd5789d2f60b4e858c9127a2e0482b10, State: Running, Role: LEADER
I20260619 10:26:45.782495 20273 consensus_queue.cc:237] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } }
I20260619 10:26:45.783282 19794 catalog_manager.cc:5495] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd5789d2f60b4e858c9127a2e0482b10 (127.9.90.66). New cstate: current_term: 1 leader_uuid: "bd5789d2f60b4e858c9127a2e0482b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd5789d2f60b4e858c9127a2e0482b10" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 } health_report { overall_health: UNKNOWN } } }
I20260619 10:26:45.795637 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.793322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.9.90.66:35789)
0619 10:26:45.793381 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:26:45.793382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:26:45.793390 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:26:45.793421 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:26:45.793422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:26:45.793440 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:26:45.793497 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:26:45.793642 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.794203 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.794925 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:26:45.795118 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:26:45.795171 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:26:45.795320 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:26:45.795417 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:26:45.795486 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:26:45.795530 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 10:26:45.795549 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 10:26:45.797261 20207 raft_consensus.cc:1261] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [term 1 FOLLOWER]: Refusing update from remote peer bd5789d2f60b4e858c9127a2e0482b10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:45.797262 19939 raft_consensus.cc:1261] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59 [term 1 FOLLOWER]: Refusing update from remote peer bd5789d2f60b4e858c9127a2e0482b10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:26:45.797577 20273 consensus_queue.cc:1035] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36608394994d4e3fb198f11c26360acb" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:45.797704 20277 consensus_queue.cc:1035] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d4ea69aeb84d58af03128ab5286f59" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:26:45.800062 20281 mvcc.cc:205] Tried to move back new op lower bound from 7298518244543102976 to 7298518244485648384. Current Snapshot: MvccSnapshot[applied={T|T < 7298518244543102976}]
I20260619 10:26:45.800434 20280 mvcc.cc:205] Tried to move back new op lower bound from 7298518244543102976 to 7298518244485648384. Current Snapshot: MvccSnapshot[applied={T|T < 7298518244543102976}]
I20260619 10:26:45.802801 19794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:45.802881 19794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:26:45.803573 19794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a57eedcff9574dbd93ea0fc81815ac35: Sending DeleteTablet for 3 replicas of tablet 8fc618f99fe24bd4b7058a6ef01caea6
I20260619 10:26:45.803889 20187 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc618f99fe24bd4b7058a6ef01caea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:45 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:60326
I20260619 10:26:45.803894 19919 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc618f99fe24bd4b7058a6ef01caea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:45 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:55668
I20260619 10:26:45.803922 20053 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc618f99fe24bd4b7058a6ef01caea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:26:45 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:40420
I20260619 10:26:45.804148 20292 tablet_replica.cc:330] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59: stopping tablet replica
I20260619 10:26:45.804157 20291 tablet_replica.cc:330] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: stopping tablet replica
I20260619 10:26:45.804253 20291 raft_consensus.cc:2229] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:26:45.804265 20292 raft_consensus.cc:2229] T 8fc618f99fe24bd4b7058a6ef01caea6 P d1d4ea69aeb84d58af03128ab5286f59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:45.804343 20291 raft_consensus.cc:2258] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:45.804507 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 19856
I20260619 10:26:45.805370 20291 ts_tablet_manager.cc:1905] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:45.806042 20290 tablet_replica.cc:330] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: stopping tablet replica
I20260619 10:26:45.806134 20290 raft_consensus.cc:2229] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:26:45.806224 20290 raft_consensus.cc:2258] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:26:45.806643 20290 ts_tablet_manager.cc:1905] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:26:45.807070 20291 ts_tablet_manager.cc:1918] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:45.807147 20291 log.cc:1196] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/8fc618f99fe24bd4b7058a6ef01caea6
I20260619 10:26:45.807405 20291 ts_tablet_manager.cc:1939] T 8fc618f99fe24bd4b7058a6ef01caea6 P bd5789d2f60b4e858c9127a2e0482b10: Deleting consensus metadata
I20260619 10:26:45.807797 19779 catalog_manager.cc:4841] TS bd5789d2f60b4e858c9127a2e0482b10 (127.9.90.66:35789): tablet 8fc618f99fe24bd4b7058a6ef01caea6 (table test-table [id=4342bec7b9fe44b7afaf7c5fd8f08d60]) successfully deleted
I20260619 10:26:45.808213 20290 ts_tablet_manager.cc:1918] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:26:45.808269 20290 log.cc:1196] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/8fc618f99fe24bd4b7058a6ef01caea6
I20260619 10:26:45.808594 20290 ts_tablet_manager.cc:1939] T 8fc618f99fe24bd4b7058a6ef01caea6 P 36608394994d4e3fb198f11c26360acb: Deleting consensus metadata
I20260619 10:26:45.808956 19778 catalog_manager.cc:4841] TS 36608394994d4e3fb198f11c26360acb (127.9.90.67:35431): tablet 8fc618f99fe24bd4b7058a6ef01caea6 (table test-table [id=4342bec7b9fe44b7afaf7c5fd8f08d60]) successfully deleted
W20260619 10:26:45.811003 19779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:37735 (error 108)
I20260619 10:26:45.811497 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 19990
I20260619 10:26:45.812362 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:26:45.812245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:37735 (local address 127.0.0.1:55674)
0619 10:26:45.812297 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:26:45.812332 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:37735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 10:26:45.812474 19779 catalog_manager.cc:4584] TS d1d4ea69aeb84d58af03128ab5286f59 (127.9.90.65:37735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fc618f99fe24bd4b7058a6ef01caea6: Network error: Client connection negotiation failed: client connection to 127.9.90.65:37735: connect: Connection refused (error 111)
I20260619 10:26:45.816437 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 20124
I20260619 10:26:45.820021 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 19762
2026-06-19T10:26:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1796 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[20296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[20296](info): set up 2 sockets
Jun 19 10:26:45 dist-test-slave-hw7q krb5kdc[20296](info): commencing operation
krb5kdc: starting...
W20260619 10:26:46.762328 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:26:46 dist-test-slave-hw7q krb5kdc[20296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:26:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:26:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/rangeradmin_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/rangeradmin_127.9.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/rangerlookup_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/rangerlookup_127.9.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/HTTP_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/HTTP_127.9.90.84@KRBTEST.COM.keytab.
I20260619 10:26:46.903718 9577 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzGZPou/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/postgres -l logfile start
2026-06-19 10:26:49.104 UTC [20326] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 10:26:49.104 UTC [20326] LOG: listening on IPv4 address "127.9.90.84", port 37365
2026-06-19 10:26:49.107 UTC [20326] LOG: listening on Unix socket "/tmp/.s.PGSQL.37365"
2026-06-19 10:26:49.134 UTC [20329] LOG: database system was shut down at 2026-06-19 10:26:47 UTC
2026-06-19 10:26:49.138 UTC [20326] LOG: database system is ready to accept connections
I20260619 10:26:50.006299 9577 mini_postgres.cc:96] Postgres bound to 37365
2026-06-19 10:26:50.011 UTC [20337] FATAL: database "slave" does not exist
127.9.90.84:37365 - accepting connections
I20260619 10:26:50.012351 9577 mini_ranger.cc:161] Starting Ranger...
I20260619 10:26:50.026182 9577 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 10:26:50.411248 9577 mini_ranger.cc:87] Created ranger Postgres database
I20260619 10:26:50.411324 9577 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-admin
2026-06-19 10:26:50,687 [I] DB FLAVOR :POSTGRES
2026-06-19 10:26:50,687 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 10:26:50,688 [I] Checking connection..
2026-06-19 10:26:50,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 10:26:50,928 [I] Checking connection passed.
2026-06-19 10:26:50,928 [I] --------- Verifying version history table ---------
2026-06-19 10:26:50,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:26:51,159 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 10:26:51,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:26:51,412 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 10:26:51,412 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 10:26:51,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 10:26:51.657 UTC [20401] WARNING: there is no transaction in progress
2026-06-19 10:26:51,662 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 10:26:51,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:26:51,910 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 10:26:51,910 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 10:26:51,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 10:26:52,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 10:26:52,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hw7q.c.gcp-upstream.internal','N') ;"
2026-06-19 10:26:52,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 10:26:52,850 [I] Table x_portal_user does not exist in database ranger
2026-06-19 10:26:52,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 10:26:53,077 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 10:26:53,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 10:26:53,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 10:26:53,554 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 10:26:53,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 10:26:54.439 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.454 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.469 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.487 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.503 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.520 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.570 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.576 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.588 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.597 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.607 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.619 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.628 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.633 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.639 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:54.645 UTC [20509] WARNING: there is no transaction in progress
2026-06-19 10:26:55,207 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 10:26:55,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 10:26:55,448 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 10:26:55,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 10:26:55,667 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 10:26:55,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 10:26:55,894 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 10:26:55,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 10:26:56,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 10:26:56,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hw7q.c.gcp-upstream.internal';"
2026-06-19 10:26:56,588 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 10:26:56,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 10:26:56,821 [I] DB_PATCHES have already been applied
I20260619 10:26:56.826328 9577 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzGZPou/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 10:26:56.826390 9577 mini_ranger.cc:193] Using host: 127.9.90.84
I20260619 10:26:56.829071 9577 mini_ranger.cc:227] Ranger admin URL: http://127.9.90.84:43235
Jun 19, 2026 10:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 10:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 10:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 10:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 10:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 10:26:58.877058 9577 mini_ranger.cc:160] Time spent starting Ranger: real 8.865s user 0.002s sys 0.009s
Jun 19, 2026 10:27:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 10:27:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 10:27:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 10:27:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 10:27:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 10:27:18.341471 9577 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/rangerkms_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/rangerkms_127.9.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.90.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.90.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.90.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/HTTP_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.90.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/HTTP_127.9.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 10:27:18 dist-test-slave-hw7q krb5kdc[20296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864838, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 10:27:18.431105 9577 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 10:27:18.439960 9577 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 10:27:19.167699 9577 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 10:27:19.167789 9577 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-kms
I20260619 10:27:19.167809 9577 mini_ranger_kms.cc:227] Using postgres at 127.9.90.84:37365
2026-06-19 10:27:19,324 [I] DB FLAVOR :POSTGRES
2026-06-19 10:27:19,324 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 10:27:19,324 [I] Checking connection
2026-06-19 10:27:19,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 10:27:19,547 [I] connection success
2026-06-19 10:27:19,548 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 10:27:19,548 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 10:27:19,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 10:27:19,769 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 10:27:19,769 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 10:27:19,770 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 10:27:19,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37365/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 10:27:20,012 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 10:27:20.461045 9577 mini_ranger_kms.cc:310] Created kms service
I20260619 10:27:20.660801 9577 mini_ranger_kms.cc:326] Created kudu user
I20260619 10:27:20.702150 9577 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 10:27:20 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864838, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:27:21.103708 9577 mini_ranger_kms.cc:384] Added ranger policy
I20260619 10:27:21.103832 9577 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 10:27:21.103856 9577 mini_ranger_kms.cc:240] Using host: 127.9.90.84
I20260619 10:27:21.106941 9577 mini_ranger_kms.cc:276] Ranger KMS PID: 20690
I20260619 10:27:21.107055 9577 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.90.84:45723
Jun 19, 2026 10:27:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:27:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:27:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 10:27:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 10:27:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 10:27:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 10:27:23.099340 9577 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.668s user 0.006s sys 0.133s
I20260619 10:27:23.099427 9577 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 10:27:23.099506 9577 mini_ranger_kms.cc:401] 127.9.90.84:45723/kms/v1/keys
Jun 19 10:27:23 dist-test-slave-hw7q krb5kdc[20296](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.90.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 10:27:23 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.90.84@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
Jun 19 10:27:24 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.90.84@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
Jun 19 10:27:26 dist-test-slave-hw7q krb5kdc[20296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:27:26.254647 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42341
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:42341
--ranger_config_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:45723/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:27:26.333113 20741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:27:26.333316 20741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:27:26.333345 20741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:27:26.334503 20741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:27:26.334542 20741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:27:26.334561 20741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:27:26.334576 20741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:27:26.334592 20741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:27:26.336131 20741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.90.84:45723/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:42341
--ranger_config_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:27:26.336366 20741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:27:26.336560 20741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:27:26.342183 20747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:27:26.342182 20749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:27:26.342398 20741 server_base.cc:1034] running on GCE node
W20260619 10:27:26.342540 20746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:27:26.342787 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:27:26.343125 20741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:27:26.344553 20741 hybrid_clock.cc:648] HybridClock initialized: now 1781864846344539 us; error 37 us; skew 500 ppm
Jun 19 10:27:26 dist-test-slave-hw7q krb5kdc[20296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864846, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:27:26.347419 20741 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:27:26.347836 20741 webserver.cc:460] Webserver started at http://127.9.90.126:42875/ using document root <none> and password file <none>
I20260619 10:27:26.348027 20741 fs_manager.cc:359] Metadata directory not provided
I20260619 10:27:26.348064 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:27:26.348177 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:27:26 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864846, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:27:26.408871 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "03beb46d8fca482b96ac4df4fddd2b60"
format_stamp: "Formatted at 2026-06-19 10:27:26 on dist-test-slave-hw7q"
server_key: "2f971aea5bd90698aec860e65ae38674"
server_key_iv: "e843d7657c5107eb77acdadf56a15d2a"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:26.409296 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "03beb46d8fca482b96ac4df4fddd2b60"
format_stamp: "Formatted at 2026-06-19 10:27:26 on dist-test-slave-hw7q"
server_key: "2f971aea5bd90698aec860e65ae38674"
server_key_iv: "e843d7657c5107eb77acdadf56a15d2a"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:26.410651 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:27:26.424144 20756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:26.424559 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:27:26.424666 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "03beb46d8fca482b96ac4df4fddd2b60"
format_stamp: "Formatted at 2026-06-19 10:27:26 on dist-test-slave-hw7q"
server_key: "2f971aea5bd90698aec860e65ae38674"
server_key_iv: "e843d7657c5107eb77acdadf56a15d2a"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:26.424762 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:27:26.454106 20741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:27:26.456115 20741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:27:26.456276 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:27:26.461006 20741 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:42341
I20260619 10:27:26.461035 20808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:42341 every 8 connection(s)
I20260619 10:27:26.461505 20741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:27:26.462710 20809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:27:26.464848 20809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60: Bootstrap starting.
I20260619 10:27:26.465366 20809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60: Neither blocks nor log segments found. Creating new log.
I20260619 10:27:26.465615 20809 log.cc:824] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60: Log is configured to *not* fsync() on all Append() calls
I20260619 10:27:26.466259 20809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60: No bootstrap required, opened a new log
I20260619 10:27:26.468169 20809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03beb46d8fca482b96ac4df4fddd2b60" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42341 } }
I20260619 10:27:26.468314 20809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:27:26.468348 20809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03beb46d8fca482b96ac4df4fddd2b60, State: Initialized, Role: FOLLOWER
I20260619 10:27:26.468494 20809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03beb46d8fca482b96ac4df4fddd2b60" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42341 } }
I20260619 10:27:26.468568 20809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:27:26.468598 20809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:27:26.468645 20809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:27:26.469161 20809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03beb46d8fca482b96ac4df4fddd2b60" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42341 } }
I20260619 10:27:26.469309 20809 leader_election.cc:302] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03beb46d8fca482b96ac4df4fddd2b60; no voters:
I20260619 10:27:26.469506 20809 leader_election.cc:288] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:27:26.469544 20812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:27:26.469720 20812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [term 1 LEADER]: Becoming Leader. State: Replica: 03beb46d8fca482b96ac4df4fddd2b60, State: Running, Role: LEADER
I20260619 10:27:26.469905 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 20741
I20260619 10:27:26.469919 20812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03beb46d8fca482b96ac4df4fddd2b60" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42341 } }
I20260619 10:27:26.469996 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:27:26.470172 9577 external_mini_cluster.cc:1428] Setting key 05bd30c071f32cb284e24acc70c9ac5e
I20260619 10:27:26.470270 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03beb46d8fca482b96ac4df4fddd2b60. Latest consensus state: current_term: 1 leader_uuid: "03beb46d8fca482b96ac4df4fddd2b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03beb46d8fca482b96ac4df4fddd2b60" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42341 } } }
I20260619 10:27:26.470331 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [sys.catalog]: This master's current role is: LEADER
I20260619 10:27:26.470778 20809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:27:26.470942 20814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03beb46d8fca482b96ac4df4fddd2b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03beb46d8fca482b96ac4df4fddd2b60" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42341 } } }
I20260619 10:27:26.471197 20814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60 [sys.catalog]: This master's current role is: LEADER
I20260619 10:27:26.471568 20818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:27:26.472147 20818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:27:26.472481 20809 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 10:27:26.480362 20818 catalog_manager.cc:1347] Generated new cluster ID: 8ba3209f943042c6b97b172ae138da7d
I20260619 10:27:26.480468 20818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:27:26 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:27:26.486405 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:26.470611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.9.90.126:42341)
0619 10:27:26.470758 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:26.470768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:26.471387 (+ 619us) server_negotiation.cc:407] Connection header received
0619 10:27:26.471529 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:26.471537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:26.471597 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:26.471739 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:27:26.473576 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:26.477524 (+ 3948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:26.478633 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:26.479383 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:26.479506 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:26.482131 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:27:26.482135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:27:26.482135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:27:26.482147 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:27:26.484644 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:26.485001 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:26.485003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:26.485004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:26.485096 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:26.485270 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:26.485271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:26.485271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:26.485544 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:27:26.485994 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:26.486062 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 10:27:26.486176 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260619 10:27:26.489622 20818 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:27:26.490445 20818 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:27:26.501256 20818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60: Generated new TSK 0
I20260619 10:27:26.501518 20818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:27:27.240402 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:45723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42341
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:27:27.324779 20854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:27:27.324967 20854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:27:27.324996 20854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:27:27.326128 20854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:27:27.326165 20854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:27:27.326198 20854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:27:27.327633 20854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.90.84:45723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:27:27.327859 20854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:27:27.328050 20854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:27:27.332845 20859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:27:27.332856 20860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:27:27.333156 20854 server_base.cc:1034] running on GCE node
W20260619 10:27:27.333179 20862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:27:27.333503 20854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:27:27.333756 20854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:27:27.334968 20854 hybrid_clock.cc:648] HybridClock initialized: now 1781864847334946 us; error 41 us; skew 500 ppm
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:27:27.337985 20854 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:27:27.338497 20854 webserver.cc:460] Webserver started at http://127.9.90.65:39531/ using document root <none> and password file <none>
I20260619 10:27:27.338775 20854 fs_manager.cc:359] Metadata directory not provided
I20260619 10:27:27.338846 20854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:27:27.338975 20854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:27:27.355548 20854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "fa861a8e6cc5437da287dc5a640f4f62"
format_stamp: "Formatted at 2026-06-19 10:27:27 on dist-test-slave-hw7q"
server_key: "fa2ba13b83dc37cb60ae6cdf2127c1df"
server_key_iv: "37cbb2ee8e049510d52a7acfcee36502"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:27.356061 20854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "fa861a8e6cc5437da287dc5a640f4f62"
format_stamp: "Formatted at 2026-06-19 10:27:27 on dist-test-slave-hw7q"
server_key: "fa2ba13b83dc37cb60ae6cdf2127c1df"
server_key_iv: "37cbb2ee8e049510d52a7acfcee36502"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:27.357990 20854 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 10:27:27.370012 20869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:27.370213 20854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:27:27.370316 20854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "fa861a8e6cc5437da287dc5a640f4f62"
format_stamp: "Formatted at 2026-06-19 10:27:27 on dist-test-slave-hw7q"
server_key: "fa2ba13b83dc37cb60ae6cdf2127c1df"
server_key_iv: "37cbb2ee8e049510d52a7acfcee36502"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:27.370424 20854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:27:27.383312 20854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:27:27.385520 20854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:27:27.385748 20854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:27:27.386179 20854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:27:27.386622 20854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:27:27.386713 20854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:27.386814 20854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:27:27.386893 20854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:27.394176 20854 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:38523
I20260619 10:27:27.394249 20983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:38523 every 8 connection(s)
I20260619 10:27:27.394603 20854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:27:27.395284 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 20854
I20260619 10:27:27.395375 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:27:27.395571 9577 external_mini_cluster.cc:1428] Setting key d0018b11a9f61de14a8446f50b0debf5
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:27:27.406255 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.395328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42341 (local address 127.9.90.65:52427)
0619 10:27:27.395994 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:27.396041 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 10:27:27.396657 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:27:27.397092 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:27:27.397098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:27:27.397343 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:27:27.397616 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.397623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.398656 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.398657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.399819 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.399822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.400321 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.400322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.400461 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.401023 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:27:27.401033 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:27:27.402649 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:27:27.404613 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:27:27.404620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:27:27.404623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:27:27.404902 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:27:27.405099 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:27:27.405100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:27:27.405101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:27:27.405220 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:27:27.405512 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:27:27.405519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:27:27.405696 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 10:27:27.405864 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 10:27:27.406021 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":94,"threads_started":1}
I20260619 10:27:27.406651 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.395636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:52427 (local address 127.9.90.126:42341)
0619 10:27:27.396849 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:27.396851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:27.396859 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:27:27.396884 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:27.396884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:27.396896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:27.396957 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:27:27.397729 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.398547 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.399912 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.400183 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.400242 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.402776 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:27:27.402778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:27:27.402779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:27:27.402787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:27:27.404524 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:27.404978 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:27.404979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:27.404979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:27.405042 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:27.405283 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:27.405284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:27.405285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:27.405427 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:27:27.405487 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:27.406478 (+ 991us) server_negotiation.cc:299] Negotiation successful
0619 10:27:27.406559 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":95,"threads_started":1}
I20260619 10:27:27.407274 20984 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42341
I20260619 10:27:27.407382 20984 heartbeater.cc:464] Registering TS with master...
I20260619 10:27:27.407619 20984 heartbeater.cc:511] Master 127.9.90.126:42341 requested a full tablet report, sending...
I20260619 10:27:27.408191 20773 ts_manager.cc:194] Registered new tserver with Master: fa861a8e6cc5437da287dc5a640f4f62 (127.9.90.65:38523)
I20260619 10:27:27.409065 20773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:52427
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:27:27.438313 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:45723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42341
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:27:27.546268 20991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:27:27.546531 20991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:27:27.546563 20991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:27:27.548233 20991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:27:27.548287 20991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:27:27.548326 20991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:27:27.550446 20991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.90.84:45723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:27:27.550689 20991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:27:27.550982 20991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:27:27.556845 21001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:27:27.556859 20999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:27:27.557165 20998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:27:27.557967 20991 server_base.cc:1034] running on GCE node
I20260619 10:27:27.558195 20991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:27:27.558452 20991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:27:27.559617 20991 hybrid_clock.cc:648] HybridClock initialized: now 1781864847559594 us; error 37 us; skew 500 ppm
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:27:27.562922 20991 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:27:27.563442 20991 webserver.cc:460] Webserver started at http://127.9.90.66:33049/ using document root <none> and password file <none>
I20260619 10:27:27.563697 20991 fs_manager.cc:359] Metadata directory not provided
I20260619 10:27:27.563832 20991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:27:27.564325 20991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:27:27.585574 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "a07e193b57314f81a9eae1649c40e11a"
format_stamp: "Formatted at 2026-06-19 10:27:27 on dist-test-slave-hw7q"
server_key: "1c83d6967220727db759f63beda5ff7b"
server_key_iv: "3d2e14cec99120d26fd42b87b7a52121"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:27.586133 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "a07e193b57314f81a9eae1649c40e11a"
format_stamp: "Formatted at 2026-06-19 10:27:27 on dist-test-slave-hw7q"
server_key: "1c83d6967220727db759f63beda5ff7b"
server_key_iv: "3d2e14cec99120d26fd42b87b7a52121"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:27.590167 20991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.001s
I20260619 10:27:27.609901 21008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:27.610087 20991 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260619 10:27:27.610179 20991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "a07e193b57314f81a9eae1649c40e11a"
format_stamp: "Formatted at 2026-06-19 10:27:27 on dist-test-slave-hw7q"
server_key: "1c83d6967220727db759f63beda5ff7b"
server_key_iv: "3d2e14cec99120d26fd42b87b7a52121"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:27.610277 20991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:27:27.624102 20991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:27:27.626384 20991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:27:27.626591 20991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:27:27.626837 20991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:27:27.627231 20991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:27:27.627331 20991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:27.627403 20991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:27:27.627456 20991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:27.635978 20991 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:37985
I20260619 10:27:27.636464 20991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:27:27.637059 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 20991
I20260619 10:27:27.637159 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:27:27.637334 9577 external_mini_cluster.cc:1428] Setting key 36a9fcbc580a58579d73dc11c78fd551
I20260619 10:27:27.638092 21121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:37985 every 8 connection(s)
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:27:27.648358 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.637999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:38535 (local address 127.9.90.126:42341)
0619 10:27:27.638049 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:27.638051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:27.638400 (+ 349us) server_negotiation.cc:407] Connection header received
0619 10:27:27.639058 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:27.639059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:27.639070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:27.639100 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:27:27.639919 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.640691 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.642200 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.642523 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.642566 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.645102 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:27:27.645104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:27:27.645104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:27:27.645112 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:27:27.646820 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:27.647264 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:27.647265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:27.647266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:27.647336 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:27.647602 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:27.647603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:27.647603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:27.647736 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:27:27.647778 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:27.648160 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 10:27:27.648247 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:27:27.649170 21124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.637698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42341 (local address 127.9.90.66:38535)
0619 10:27:27.638267 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:27.638319 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 10:27:27.638937 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:27:27.639298 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:27:27.639302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:27:27.639541 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:27:27.639806 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.639812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.640803 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.640804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.642103 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.642106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.642652 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.642654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.642801 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.643343 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:27:27.643353 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:27:27.644993 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:27:27.646909 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:27:27.646916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:27:27.646918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:27:27.647183 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:27:27.647403 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:27:27.647404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:27:27.647404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:27:27.647531 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:27:27.647826 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:27:27.647832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:27:27.648005 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 10:27:27.648767 (+ 762us) client_negotiation.cc:240] Negotiation successful
0619 10:27:27.648932 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":110,"threads_started":1}
I20260619 10:27:27.649832 21122 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42341
I20260619 10:27:27.649921 21122 heartbeater.cc:464] Registering TS with master...
I20260619 10:27:27.650098 21122 heartbeater.cc:511] Master 127.9.90.126:42341 requested a full tablet report, sending...
I20260619 10:27:27.650450 20773 ts_manager.cc:194] Registered new tserver with Master: a07e193b57314f81a9eae1649c40e11a (127.9.90.66:37985)
I20260619 10:27:27.650928 20773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:38535
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:27:27.677120 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:45723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42341
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:27:27.788157 21128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:27:27.788367 21128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:27:27.788398 21128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:27:27.789518 21128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:27:27.789558 21128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:27:27.789592 21128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:27:27.791025 21128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.90.84:45723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:27:27.791239 21128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:27:27.791462 21128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:27:27.796828 21134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:27:27.797051 21133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:27:27.797998 21128 server_base.cc:1034] running on GCE node
W20260619 10:27:27.798224 21136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:27:27.798453 21128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:27:27.798695 21128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:27:27.799841 21128 hybrid_clock.cc:648] HybridClock initialized: now 1781864847799824 us; error 30 us; skew 500 ppm
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:27:27.802356 21128 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:27:27.802788 21128 webserver.cc:460] Webserver started at http://127.9.90.67:43147/ using document root <none> and password file <none>
I20260619 10:27:27.803006 21128 fs_manager.cc:359] Metadata directory not provided
I20260619 10:27:27.803052 21128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:27:27.803171 21128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:27:27.815649 21128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "ec5a533c554c4849967c32dca7182924"
format_stamp: "Formatted at 2026-06-19 10:27:27 on dist-test-slave-hw7q"
server_key: "206c7adcfb279dbbd0eef08d6e8646d4"
server_key_iv: "63900988cb01424a6b94396b05ce6797"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:27.816177 21128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "ec5a533c554c4849967c32dca7182924"
format_stamp: "Formatted at 2026-06-19 10:27:27 on dist-test-slave-hw7q"
server_key: "206c7adcfb279dbbd0eef08d6e8646d4"
server_key_iv: "63900988cb01424a6b94396b05ce6797"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:27.826241 21128 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260619 10:27:27.841017 21143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:27.841503 21128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 10:27:27.841656 21128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "ec5a533c554c4849967c32dca7182924"
format_stamp: "Formatted at 2026-06-19 10:27:27 on dist-test-slave-hw7q"
server_key: "206c7adcfb279dbbd0eef08d6e8646d4"
server_key_iv: "63900988cb01424a6b94396b05ce6797"
server_key_version: "kuduclusterkey@0"
I20260619 10:27:27.841890 21128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:27:27.867901 21128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:27:27.870101 21128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:27:27.870256 21128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:27:27.870510 21128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:27:27.870852 21128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:27:27.870898 21128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:27.870934 21128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:27:27.870960 21128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:27:27.877614 21128 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:46105
I20260619 10:27:27.877898 21256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:46105 every 8 connection(s)
I20260619 10:27:27.878046 21128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:27:27 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:27:27.887342 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.878840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:41125 (local address 127.9.90.126:42341)
0619 10:27:27.878890 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:27.878892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:27.879098 (+ 206us) server_negotiation.cc:407] Connection header received
0619 10:27:27.879609 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:27.879610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:27.879622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:27.879699 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:27:27.880235 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.880964 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.882121 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.882383 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.882451 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.884624 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:27:27.884626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:27:27.884626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:27:27.884634 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:27:27.886059 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:27.886421 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:27.886422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:27.886422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:27.886488 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:27.886701 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:27.886702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:27.886702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:27.886855 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:27:27.886938 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:27.887158 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 10:27:27.887244 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:27:27.887490 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.878813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42341 (local address 127.9.90.67:41125)
0619 10:27:27.878970 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:27.879013 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 10:27:27.879515 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:27:27.879755 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:27:27.879759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:27:27.879942 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:27:27.880141 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.880146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.881070 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.881071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.882027 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.882031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.882477 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.882478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.882590 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.883141 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:27:27.883150 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:27:27.884535 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:27:27.886132 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:27:27.886137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:27:27.886139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:27:27.886354 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:27:27.886550 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:27:27.886550 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:27:27.886550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:27:27.886638 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:27:27.886920 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:27:27.886923 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:27:27.887044 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 10:27:27.887128 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 10:27:27.887279 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260619 10:27:27.887507 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 21128
I20260619 10:27:27.887593 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:27:27.887789 9577 external_mini_cluster.cc:1428] Setting key 0a4650f6d10db791fac4daa744ac6cfe
I20260619 10:27:27.888118 21257 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42341
I20260619 10:27:27.888258 21257 heartbeater.cc:464] Registering TS with master...
I20260619 10:27:27.888445 21257 heartbeater.cc:511] Master 127.9.90.126:42341 requested a full tablet report, sending...
I20260619 10:27:27.888811 20773 ts_manager.cc:194] Registered new tserver with Master: ec5a533c554c4849967c32dca7182924 (127.9.90.67:46105)
I20260619 10:27:27.889258 20773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:41125
I20260619 10:27:27.899509 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:27:27.909152 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.901169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.9.90.126:42341)
0619 10:27:27.901215 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:27.901217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:27.901494 (+ 277us) server_negotiation.cc:407] Connection header received
0619 10:27:27.901603 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:27.901603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:27.901615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:27.901643 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:27:27.902381 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.903163 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.904255 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.904548 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.904587 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.906153 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:27:27.906154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:27:27.906155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:27:27.906162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:27:27.907894 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:27.908306 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:27.908307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:27.908307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:27.908373 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:27:27.908572 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:27:27.908573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:27:27.908573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:27:27.908696 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:27:27.908771 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:27.908978 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 10:27:27.909055 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:27:27.911448 20773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:27:27.913620 20773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:27:27.921372 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.916022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:46105 (local address 127.0.0.1:41208)
0619 10:27:27.916955 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:27.916964 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:27:27.917003 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:27:27.917249 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:27:27.917252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:27:27.917254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:27:27.917321 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.917323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.918752 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.918753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.920058 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.920059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.921080 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.921081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.921169 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.921183 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:27:27.921246 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 10:27:27.921297 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":75,"threads_started":1}
I20260619 10:27:27.921744 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.916435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.9.90.66:37985)
0619 10:27:27.916588 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:27.916593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:27.917384 (+ 791us) server_negotiation.cc:407] Connection header received
0619 10:27:27.917491 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:27.917494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:27.917509 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:27.917558 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:27:27.917757 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.918923 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.920169 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.921191 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.921230 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.921314 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:27.921635 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 10:27:27.921673 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260619 10:27:27.921578 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.916250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:37985 (local address 127.0.0.1:38724)
0619 10:27:27.917323 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:27.917332 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:27:27.917410 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:27:27.917648 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:27:27.917649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:27:27.917650 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:27:27.917692 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.917693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.919001 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.919002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.920056 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.920058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.921366 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.921367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.921437 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.921440 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:27:27.921480 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 10:27:27.921517 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":7,"thread_start_us":61,"threads_started":1}
I20260619 10:27:27.922107 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.916271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.9.90.67:46105)
0619 10:27:27.917098 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:27.917101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:27.917108 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:27:27.917139 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:27.917141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:27.917153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:27.917217 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:27:27.917402 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.918530 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.920169 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.920901 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.920967 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.921050 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:27.921985 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 10:27:27.922026 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":66,"threads_started":1}
I20260619 10:27:27.922770 21191 tablet_service.cc:1467] Processing CreateTablet for tablet 2880353108794db7b074c4e303f28d0d (DEFAULT_TABLE table=test-table [id=ef2848c61e6a4571b55c240fc2679f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:27:27.922770 21056 tablet_service.cc:1467] Processing CreateTablet for tablet 2880353108794db7b074c4e303f28d0d (DEFAULT_TABLE table=test-table [id=ef2848c61e6a4571b55c240fc2679f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:27:27.923105 21056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880353108794db7b074c4e303f28d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:27:27.923105 21191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880353108794db7b074c4e303f28d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:27:27.927753 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.916022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:38523 (local address 127.0.0.1:44706)
0619 10:27:27.916641 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:27.916671 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 10:27:27.916747 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:27:27.924055 (+ 7308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:27:27.924056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:27:27.924057 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:27:27.924093 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.924094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.925317 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.925318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.926372 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:27.926374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.927554 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.927555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:27.927617 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.927621 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:27:27.927650 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 10:27:27.927684 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":79,"threads_started":1}
I20260619 10:27:27.928282 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:27.916428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.9.90.65:38523)
0619 10:27:27.923652 (+ 7224us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:27.923656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:27.923668 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:27:27.923943 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:27.923946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:27.923962 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:27.924153 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:27:27.924215 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.925207 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.926467 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:27.927464 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:27.928070 (+ 606us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:27.928133 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:27.928159 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 10:27:27.928199 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7170,"thread_start_us":49,"threads_started":1}
I20260619 10:27:27.929039 20918 tablet_service.cc:1467] Processing CreateTablet for tablet 2880353108794db7b074c4e303f28d0d (DEFAULT_TABLE table=test-table [id=ef2848c61e6a4571b55c240fc2679f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:27:27.929412 20918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880353108794db7b074c4e303f28d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:27:27.930162 21275 tablet_bootstrap.cc:492] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a: Bootstrap starting.
I20260619 10:27:27.931623 21275 tablet_bootstrap.cc:654] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a: Neither blocks nor log segments found. Creating new log.
I20260619 10:27:27.931907 21275 log.cc:824] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:27:27.932233 21276 tablet_bootstrap.cc:492] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: Bootstrap starting.
I20260619 10:27:27.933218 21276 tablet_bootstrap.cc:654] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: Neither blocks nor log segments found. Creating new log.
I20260619 10:27:27.933390 21278 tablet_bootstrap.cc:492] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62: Bootstrap starting.
I20260619 10:27:27.933573 21276 log.cc:824] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: Log is configured to *not* fsync() on all Append() calls
I20260619 10:27:27.934017 21278 tablet_bootstrap.cc:654] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62: Neither blocks nor log segments found. Creating new log.
I20260619 10:27:27.934340 21275 tablet_bootstrap.cc:492] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a: No bootstrap required, opened a new log
I20260619 10:27:27.934337 21278 log.cc:824] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62: Log is configured to *not* fsync() on all Append() calls
I20260619 10:27:27.934450 21275 ts_tablet_manager.cc:1397] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 10:27:27.934988 21278 tablet_bootstrap.cc:492] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62: No bootstrap required, opened a new log
I20260619 10:27:27.935047 21278 ts_tablet_manager.cc:1397] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 10:27:27.936420 21275 raft_consensus.cc:348] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } }
I20260619 10:27:27.936571 21275 raft_consensus.cc:374] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:27:27.936600 21275 raft_consensus.cc:729] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07e193b57314f81a9eae1649c40e11a, State: Initialized, Role: FOLLOWER
I20260619 10:27:27.936749 21275 consensus_queue.cc:260] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } }
I20260619 10:27:27.936990 21275 ts_tablet_manager.cc:1428] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:27:27.937107 21122 heartbeater.cc:503] Master 127.9.90.126:42341 was elected leader, sending a full tablet report...
I20260619 10:27:27.937363 21278 raft_consensus.cc:348] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } }
I20260619 10:27:27.937502 21278 raft_consensus.cc:374] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:27:27.937530 21278 raft_consensus.cc:729] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa861a8e6cc5437da287dc5a640f4f62, State: Initialized, Role: FOLLOWER
I20260619 10:27:27.937642 21278 consensus_queue.cc:260] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } }
I20260619 10:27:27.938452 21278 ts_tablet_manager.cc:1428] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:27:27.938486 21276 tablet_bootstrap.cc:492] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: No bootstrap required, opened a new log
I20260619 10:27:27.938540 21276 ts_tablet_manager.cc:1397] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 10:27:27.938611 20984 heartbeater.cc:503] Master 127.9.90.126:42341 was elected leader, sending a full tablet report...
I20260619 10:27:27.940903 21276 raft_consensus.cc:348] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } }
I20260619 10:27:27.941035 21276 raft_consensus.cc:374] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:27:27.941066 21276 raft_consensus.cc:729] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5a533c554c4849967c32dca7182924, State: Initialized, Role: FOLLOWER
I20260619 10:27:27.941191 21276 consensus_queue.cc:260] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } }
I20260619 10:27:27.941465 21257 heartbeater.cc:503] Master 127.9.90.126:42341 was elected leader, sending a full tablet report...
I20260619 10:27:27.941460 21276 ts_tablet_manager.cc:1428] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
Jun 19 10:27:28 dist-test-slave-hw7q krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:27:28.123533 21281 raft_consensus.cc:482] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:27:28.123709 21281 raft_consensus.cc:504] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } }
I20260619 10:27:28.124145 21281 leader_election.cc:288] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa861a8e6cc5437da287dc5a640f4f62 (127.9.90.65:38523), ec5a533c554c4849967c32dca7182924 (127.9.90.67:46105)
I20260619 10:27:28.128090 21124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:28.124326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:38523 (local address 127.9.90.66:55179)
0619 10:27:28.124437 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:28.124449 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:27:28.124496 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:27:28.124794 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:27:28.124795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:27:28.124797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:27:28.124866 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:28.124868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.125809 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.125811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:28.127022 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:28.127025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.127830 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.127831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:28.127908 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:28.127911 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:27:28.127960 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:27:28.128000 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 10:27:28.128088 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:28.124601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:49285 (local address 127.9.90.67:46105)
0619 10:27:28.124646 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:28.124648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:28.124811 (+ 163us) server_negotiation.cc:407] Connection header received
0619 10:27:28.124910 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:28.124911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:28.124924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:28.124956 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:27:28.125190 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.125880 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.127174 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.127713 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.127764 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:28.127775 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:28.127993 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 10:27:28.128027 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:27:28.128325 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:28.124326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:46105 (local address 127.9.90.66:49285)
0619 10:27:28.124755 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:28.124765 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:27:28.124834 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:27:28.125017 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:27:28.125018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:27:28.125019 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:27:28.125083 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:28.125084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.125995 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.125996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:28.127067 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:27:28.127068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.127830 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.127831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:27:28.127908 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:28.127912 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:27:28.128214 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 10:27:28.128255 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":16,"thread_start_us":85,"threads_started":1}
W20260619 10:27:28.128705 21258 tablet.cc:2369] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:27:28.128758 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:28.124562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:55179 (local address 127.9.90.65:38523)
0619 10:27:28.124606 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:28.124608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:28.124617 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:27:28.124645 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:28.124646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:28.124675 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:28.124737 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:27:28.124941 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.125683 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.127144 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.127728 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.127789 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:28.127798 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:28.128644 (+ 846us) server_negotiation.cc:299] Negotiation successful
0619 10:27:28.128692 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:27:28.128739 21202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880353108794db7b074c4e303f28d0d" candidate_uuid: "a07e193b57314f81a9eae1649c40e11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5a533c554c4849967c32dca7182924" is_pre_election: true
I20260619 10:27:28.128909 21202 raft_consensus.cc:2454] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07e193b57314f81a9eae1649c40e11a in term 0.
I20260619 10:27:28.129046 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880353108794db7b074c4e303f28d0d" candidate_uuid: "a07e193b57314f81a9eae1649c40e11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa861a8e6cc5437da287dc5a640f4f62" is_pre_election: true
I20260619 10:27:28.129194 20938 raft_consensus.cc:2454] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07e193b57314f81a9eae1649c40e11a in term 0.
I20260619 10:27:28.129426 21009 leader_election.cc:302] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a07e193b57314f81a9eae1649c40e11a, fa861a8e6cc5437da287dc5a640f4f62; no voters:
I20260619 10:27:28.129562 21281 raft_consensus.cc:2781] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:27:28.129601 21281 raft_consensus.cc:482] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:27:28.129623 21281 raft_consensus.cc:3037] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:27:28.130455 21281 raft_consensus.cc:504] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } }
I20260619 10:27:28.130585 21281 leader_election.cc:288] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [CANDIDATE]: Term 1 election: Requested vote from peers fa861a8e6cc5437da287dc5a640f4f62 (127.9.90.65:38523), ec5a533c554c4849967c32dca7182924 (127.9.90.67:46105)
I20260619 10:27:28.130797 21202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880353108794db7b074c4e303f28d0d" candidate_uuid: "a07e193b57314f81a9eae1649c40e11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5a533c554c4849967c32dca7182924"
I20260619 10:27:28.130829 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880353108794db7b074c4e303f28d0d" candidate_uuid: "a07e193b57314f81a9eae1649c40e11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa861a8e6cc5437da287dc5a640f4f62"
I20260619 10:27:28.130878 21202 raft_consensus.cc:3037] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:27:28.130893 20938 raft_consensus.cc:3037] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:27:28.131688 20938 raft_consensus.cc:2454] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07e193b57314f81a9eae1649c40e11a in term 1.
I20260619 10:27:28.131685 21202 raft_consensus.cc:2454] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07e193b57314f81a9eae1649c40e11a in term 1.
I20260619 10:27:28.131915 21009 leader_election.cc:302] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a07e193b57314f81a9eae1649c40e11a, fa861a8e6cc5437da287dc5a640f4f62; no voters:
I20260619 10:27:28.132037 21281 raft_consensus.cc:2781] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:27:28.132221 21281 raft_consensus.cc:686] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 1 LEADER]: Becoming Leader. State: Replica: a07e193b57314f81a9eae1649c40e11a, State: Running, Role: LEADER
I20260619 10:27:28.132320 21281 consensus_queue.cc:237] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } }
I20260619 10:27:28.133179 20773 catalog_manager.cc:5495] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a reported cstate change: term changed from 0 to 1, leader changed from <none> to a07e193b57314f81a9eae1649c40e11a (127.9.90.66). New cstate: current_term: 1 leader_uuid: "a07e193b57314f81a9eae1649c40e11a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a07e193b57314f81a9eae1649c40e11a" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 37985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 } health_report { overall_health: UNKNOWN } } }
W20260619 10:27:28.137372 21123 tablet.cc:2369] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:27:28.146409 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:28.144036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.9.90.66:37985)
0619 10:27:28.144086 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:27:28.144088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:27:28.144097 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:27:28.144172 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:27:28.144173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:27:28.144194 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:27:28.144255 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:27:28.144408 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.144865 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.145654 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:27:28.145902 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:27:28.145957 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:27:28.146102 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:27:28.146211 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:27:28.146281 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:27:28.146301 (+ 20us) server_negotiation.cc:299] Negotiation successful
0619 10:27:28.146321 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 10:27:28.147351 20985 tablet.cc:2369] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:27:28.148480 21202 raft_consensus.cc:1261] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [term 1 FOLLOWER]: Refusing update from remote peer a07e193b57314f81a9eae1649c40e11a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:27:28.148485 20938 raft_consensus.cc:1261] T 2880353108794db7b074c4e303f28d0d P fa861a8e6cc5437da287dc5a640f4f62 [term 1 FOLLOWER]: Refusing update from remote peer a07e193b57314f81a9eae1649c40e11a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:27:28.148808 21285 consensus_queue.cc:1035] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5a533c554c4849967c32dca7182924" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 46105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:27:28.148926 21281 consensus_queue.cc:1035] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa861a8e6cc5437da287dc5a640f4f62" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:27:28.151822 21291 mvcc.cc:205] Tried to move back new op lower bound from 7298518418013499392 to 7298518417950695424. Current Snapshot: MvccSnapshot[applied={T|T < 7298518418013499392}]
I20260619 10:27:28.152005 21289 mvcc.cc:205] Tried to move back new op lower bound from 7298518418013499392 to 7298518417950695424. Current Snapshot: MvccSnapshot[applied={T|T < 7298518418013499392}]
I20260619 10:27:28.153373 20773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:27:28.153457 20773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:27:28.154193 20773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03beb46d8fca482b96ac4df4fddd2b60: Sending DeleteTablet for 3 replicas of tablet 2880353108794db7b074c4e303f28d0d
I20260619 10:27:28.154744 20918 tablet_service.cc:1514] Processing DeleteTablet for tablet 2880353108794db7b074c4e303f28d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:27:28 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:44706
I20260619 10:27:28.154961 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 20854
I20260619 10:27:28.154973 21056 tablet_service.cc:1514] Processing DeleteTablet for tablet 2880353108794db7b074c4e303f28d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:27:28 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:38724
I20260619 10:27:28.155196 21191 tablet_service.cc:1514] Processing DeleteTablet for tablet 2880353108794db7b074c4e303f28d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:27:28 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:41208
I20260619 10:27:28.157895 21299 tablet_replica.cc:330] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: stopping tablet replica
I20260619 10:27:28.157996 21299 raft_consensus.cc:2229] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:27:28.158097 21299 raft_consensus.cc:2258] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:27:28.158244 21298 tablet_replica.cc:330] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a: stopping tablet replica
I20260619 10:27:28.158334 21298 raft_consensus.cc:2229] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:27:28.158429 21298 raft_consensus.cc:2258] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:27:28.158651 21299 ts_tablet_manager.cc:1905] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:27:28.158907 21298 ts_tablet_manager.cc:1905] T 2880353108794db7b074c4e303f28d0d P a07e193b57314f81a9eae1649c40e11a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:27:28.160356 21299 ts_tablet_manager.cc:1918] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:27:28.160480 21299 log.cc:1196] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/2880353108794db7b074c4e303f28d0d
I20260619 10:27:28.161064 21299 ts_tablet_manager.cc:1939] T 2880353108794db7b074c4e303f28d0d P ec5a533c554c4849967c32dca7182924: Deleting consensus metadata
I20260619 10:27:28.161527 20759 catalog_manager.cc:4841] TS ec5a533c554c4849967c32dca7182924 (127.9.90.67:46105): tablet 2880353108794db7b074c4e303f28d0d (table test-table [id=ef2848c61e6a4571b55c240fc2679f53]) successfully deleted
W20260619 10:27:28.161697 20757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:38523 (error 108)
I20260619 10:27:28.161870 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 20991
I20260619 10:27:28.166396 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 21128
I20260619 10:27:28.167375 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:28.167228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:38523 (local address 127.0.0.1:44720)
0619 10:27:28.167296 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:28.167337 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 10:27:28.167510 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:27:28.167245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:37985 (local address 127.0.0.1:38730)
0619 10:27:28.167471 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 10:27:28.167494 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.66:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
W20260619 10:27:28.167572 20757 catalog_manager.cc:4584] TS fa861a8e6cc5437da287dc5a640f4f62 (127.9.90.65:38523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2880353108794db7b074c4e303f28d0d: Network error: Client connection negotiation failed: client connection to 127.9.90.65:38523: connect: Connection refused (error 111)
I20260619 10:27:28.171926 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 20741
I20260619 10:27:28.176244 9577 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 10:27:28.510010 9577 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 10:27:28.510106 9577 mini_ranger.cc:66] Stopping Ranger...
I20260619 10:27:28.843997 9577 mini_ranger.cc:68] Stopped Ranger
2026-06-19 10:27:28.844 UTC [20326] LOG: received smart shutdown request
2026-06-19 10:27:28.847 UTC [20326] LOG: background worker "logical replication launcher" (PID 20335) exited with exit code 1
2026-06-19 10:27:28.847 UTC [20330] LOG: shutting down
2026-06-19 10:27:28.987 UTC [20326] LOG: database system is shut down
2026-06-19T10:27:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43257 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:27:29 dist-test-slave-hw7q krb5kdc[21309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:27:29 dist-test-slave-hw7q krb5kdc[21309](info): set up 2 sockets
Jun 19 10:27:29 dist-test-slave-hw7q krb5kdc[21309](info): commencing operation
krb5kdc: starting...
W20260619 10:27:30.021579 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:27:30 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:27:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/rangeradmin_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/rangeradmin_127.9.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/rangerlookup_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/rangerlookup_127.9.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/HTTP_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/HTTP_127.9.90.84@KRBTEST.COM.keytab.
I20260619 10:27:30.170311 9577 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzGZPou/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/postgres -l logfile start
2026-06-19 10:27:32.382 UTC [21339] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 10:27:32.382 UTC [21339] LOG: listening on IPv4 address "127.9.90.84", port 37793
2026-06-19 10:27:32.385 UTC [21339] LOG: listening on Unix socket "/tmp/.s.PGSQL.37793"
2026-06-19 10:27:32.411 UTC [21342] LOG: database system was shut down at 2026-06-19 10:27:31 UTC
2026-06-19 10:27:32.414 UTC [21339] LOG: database system is ready to accept connections
I20260619 10:27:33.273332 9577 mini_postgres.cc:96] Postgres bound to 37793
2026-06-19 10:27:33.278 UTC [21350] FATAL: database "slave" does not exist
127.9.90.84:37793 - accepting connections
I20260619 10:27:33.279250 9577 mini_ranger.cc:161] Starting Ranger...
I20260619 10:27:33.293567 9577 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 10:27:33.697705 9577 mini_ranger.cc:87] Created ranger Postgres database
I20260619 10:27:33.697814 9577 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-admin
2026-06-19 10:27:33,963 [I] DB FLAVOR :POSTGRES
2026-06-19 10:27:33,963 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 10:27:33,963 [I] Checking connection..
2026-06-19 10:27:33,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 10:27:34,201 [I] Checking connection passed.
2026-06-19 10:27:34,202 [I] --------- Verifying version history table ---------
2026-06-19 10:27:34,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:27:34,452 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 10:27:34,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:27:34,676 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 10:27:34,676 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 10:27:34,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 10:27:34.926 UTC [21414] WARNING: there is no transaction in progress
2026-06-19 10:27:34,932 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 10:27:34,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 10:27:35,170 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 10:27:35,170 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 10:27:35,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 10:27:35,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 10:27:35,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hw7q.c.gcp-upstream.internal','N') ;"
2026-06-19 10:27:35,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 10:27:36,125 [I] Table x_portal_user does not exist in database ranger
2026-06-19 10:27:36,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 10:27:36,362 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 10:27:36,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 10:27:36,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 10:27:36,839 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 10:27:36,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 10:27:37.716 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.730 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.745 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.759 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.775 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.791 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.840 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.846 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.859 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.868 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.878 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.891 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.900 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.905 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.911 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:37.916 UTC [21522] WARNING: there is no transaction in progress
2026-06-19 10:27:38,467 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 10:27:38,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 10:27:38,699 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 10:27:38,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 10:27:38,946 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 10:27:38,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 10:27:39,197 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 10:27:39,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 10:27:39,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 10:27:39,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hw7q.c.gcp-upstream.internal';"
2026-06-19 10:27:39,907 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 10:27:39,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 10:27:40,162 [I] DB_PATCHES have already been applied
I20260619 10:27:40.167529 9577 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzGZPou/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 10:27:40.167589 9577 mini_ranger.cc:193] Using host: 127.9.90.84
I20260619 10:27:40.170660 9577 mini_ranger.cc:227] Ranger admin URL: http://127.9.90.84:56871
Jun 19, 2026 10:27:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:27:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:27:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 10:27:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 10:27:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 10:27:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 10:27:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 10:27:41.152956 9577 mini_ranger.cc:160] Time spent starting Ranger: real 7.874s user 0.000s sys 0.008s
Jun 19, 2026 10:27:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 10:27:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 10:27:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 10:27:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 10:28:00 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 10:28:01.163455 9577 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/rangerkms_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/rangerkms_127.9.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.90.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.90.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.90.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/HTTP_127.9.90.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.90.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/HTTP_127.9.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 10:28:01 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864881, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 10:28:01.232303 9577 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 10:28:01.240726 9577 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 10:28:02.013118 9577 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 10:28:02.013182 9577 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-kms
I20260619 10:28:02.013199 9577 mini_ranger_kms.cc:227] Using postgres at 127.9.90.84:37793
2026-06-19 10:28:02,156 [I] DB FLAVOR :POSTGRES
2026-06-19 10:28:02,156 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 10:28:02,156 [I] Checking connection
2026-06-19 10:28:02,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 10:28:02,400 [I] connection success
2026-06-19 10:28:02,400 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 10:28:02,400 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 10:28:02,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 10:28:02,625 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 10:28:02,625 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 10:28:02,626 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 10:28:02,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.90.84:37793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 10:28:02,880 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 10:28:03.341958 9577 mini_ranger_kms.cc:310] Created kms service
I20260619 10:28:03.528097 9577 mini_ranger_kms.cc:326] Created kudu user
I20260619 10:28:03.567345 9577 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 10:28:03 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864881, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:03.968592 9577 mini_ranger_kms.cc:384] Added ranger policy
I20260619 10:28:03.968704 9577 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzGZPou/build/release/bin/postgresql.jar:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzGZPou/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 10:28:03.968724 9577 mini_ranger_kms.cc:240] Using host: 127.9.90.84
I20260619 10:28:03.971832 9577 mini_ranger_kms.cc:276] Ranger KMS PID: 21702
I20260619 10:28:03.971911 9577 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.90.84:47015
Jun 19, 2026 10:28:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 10:28:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 10:28:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 10:28:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 10:28:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 10:28:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 10:28:05.978922 9577 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.747s user 0.000s sys 0.125s
I20260619 10:28:05.979002 9577 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 10:28:05.979048 9577 mini_ranger_kms.cc:401] 127.9.90.84:47015/kms/v1/keys
Jun 19 10:28:06 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864886, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.90.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 10:28:06 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864886, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.90.84@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
Jun 19 10:28:06 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864886, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.90.84@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
Jun 19 10:28:08 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:28:08.942132 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:35067
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:35067
--ranger_config_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:47015/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:09.022037 21753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:09.022233 21753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:09.022253 21753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:09.023352 21753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:09.023392 21753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:09.023406 21753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:09.023416 21753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:09.023427 21753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:09.024910 21753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.90.84:47015/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:35067
--ranger_config_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:35067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:09.025122 21753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:09.025321 21753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:09.030191 21759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:09.030231 21758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:09.030414 21761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:09.030234 21753 server_base.cc:1034] running on GCE node
I20260619 10:28:09.030721 21753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:09.031059 21753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:09.032226 21753 hybrid_clock.cc:648] HybridClock initialized: now 1781864889032204 us; error 36 us; skew 500 ppm
Jun 19 10:28:09 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:09.035156 21753 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:28:09.035653 21753 webserver.cc:460] Webserver started at http://127.9.90.126:36381/ using document root <none> and password file <none>
I20260619 10:28:09.035898 21753 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:09.035974 21753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:09.036098 21753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:28:09 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:09.086899 21753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "13a64fb8c4c54a1a84aa461cae54a6f8"
format_stamp: "Formatted at 2026-06-19 10:28:09 on dist-test-slave-hw7q"
server_key: "0bbc691963bcbbb567f0a5266da12095"
server_key_iv: "1e689e000d091a496ac7befa6a1f0309"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:09.087256 21753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "13a64fb8c4c54a1a84aa461cae54a6f8"
format_stamp: "Formatted at 2026-06-19 10:28:09 on dist-test-slave-hw7q"
server_key: "0bbc691963bcbbb567f0a5266da12095"
server_key_iv: "1e689e000d091a496ac7befa6a1f0309"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:09.088523 21753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:28:09.103837 21768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:09.104087 21753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:09.104190 21753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "13a64fb8c4c54a1a84aa461cae54a6f8"
format_stamp: "Formatted at 2026-06-19 10:28:09 on dist-test-slave-hw7q"
server_key: "0bbc691963bcbbb567f0a5266da12095"
server_key_iv: "1e689e000d091a496ac7befa6a1f0309"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:09.104286 21753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:09.118069 21753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:09.119983 21753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:09.120128 21753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:09.124539 21753 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:35067
I20260619 10:28:09.124614 21820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:35067 every 8 connection(s)
I20260619 10:28:09.124926 21753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:28:09.125568 21821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:09.127190 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 21753
I20260619 10:28:09.127277 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:28:09.127468 9577 external_mini_cluster.cc:1428] Setting key 219643334996919f4dda8f0c478b0abf
I20260619 10:28:09.127939 21821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Bootstrap starting.
I20260619 10:28:09.128588 21821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:09.128854 21821 log.cc:824] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:09.129585 21821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: No bootstrap required, opened a new log
I20260619 10:28:09.132012 21821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } }
I20260619 10:28:09.132187 21821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:09.132233 21821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a64fb8c4c54a1a84aa461cae54a6f8, State: Initialized, Role: FOLLOWER
I20260619 10:28:09.132357 21821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } }
I20260619 10:28:09.132429 21821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:28:09.132458 21821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:28:09.132496 21821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:09.133118 21821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } }
I20260619 10:28:09.133248 21821 leader_election.cc:302] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13a64fb8c4c54a1a84aa461cae54a6f8; no voters:
I20260619 10:28:09.133410 21821 leader_election.cc:288] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:28:09.133451 21826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 1 FOLLOWER]: Leader election won for term 1
Jun 19 10:28:09 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:09.133656 21826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13a64fb8c4c54a1a84aa461cae54a6f8, State: Running, Role: LEADER
I20260619 10:28:09.133734 21826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } }
I20260619 10:28:09.133772 21821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:09.134099 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } } }
I20260619 10:28:09.134678 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:09.134342 21828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13a64fb8c4c54a1a84aa461cae54a6f8. Latest consensus state: current_term: 1 leader_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } } }
I20260619 10:28:09.134799 21828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:09.135391 21821 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 10:28:09.138303 21836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:28:09.139571 21836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:28:09.140415 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:09.127999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.9.90.126:35067)
0619 10:28:09.128195 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:09.128205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:09.128237 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:28:09.128797 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:09.128806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:09.128870 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:09.129043 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:09.129658 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:09.130574 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:09.131313 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:09.131716 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:09.131815 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:09.134226 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:09.134230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:09.134230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:09.134241 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:09.137904 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:09.138385 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:09.138393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:09.138393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:09.138870 (+ 477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:09.139109 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:09.139110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:09.139110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:09.139452 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:09.139627 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:09.140036 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 10:28:09.140193 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
I20260619 10:28:09.142198 21836 catalog_manager.cc:1347] Generated new cluster ID: 15181af9467c4161ab47b5137f086cfd
I20260619 10:28:09.142248 21836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:28:09.154284 21836 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:28:09.155061 21836 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:28:09.160209 21836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Generated new TSK 0
I20260619 10:28:09.160748 21836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:28:09.822278 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:47015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:35067
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:09.932730 21866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:09.932978 21866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:09.933009 21866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:09.934664 21866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:09.934755 21866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:09.934809 21866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:28:09.936884 21866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.90.84:47015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:35067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:09.937235 21866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:09.937476 21866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:28:09.943449 21866 server_base.cc:1034] running on GCE node
W20260619 10:28:09.943488 21874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:09.943593 21871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:09.943632 21872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:09.943847 21866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:09.944056 21866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:09.945194 21866 hybrid_clock.cc:648] HybridClock initialized: now 1781864889945180 us; error 29 us; skew 500 ppm
Jun 19 10:28:09 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:09.951778 21866 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:28:09.952334 21866 webserver.cc:460] Webserver started at http://127.9.90.65:35569/ using document root <none> and password file <none>
I20260619 10:28:09.952591 21866 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:09.952664 21866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:09.952797 21866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:28:09 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:09.966027 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "5468e3583c4249db8cbc34fc867452c0"
format_stamp: "Formatted at 2026-06-19 10:28:09 on dist-test-slave-hw7q"
server_key: "077231f01a6ab758b0fc42597db0610a"
server_key_iv: "b5a478c212f18af9bf4bc5c6e5cbe290"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:09.966564 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "5468e3583c4249db8cbc34fc867452c0"
format_stamp: "Formatted at 2026-06-19 10:28:09 on dist-test-slave-hw7q"
server_key: "077231f01a6ab758b0fc42597db0610a"
server_key_iv: "b5a478c212f18af9bf4bc5c6e5cbe290"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:09.968313 21866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:09.980988 21883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:09.981284 21866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 10:28:09.981398 21866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "5468e3583c4249db8cbc34fc867452c0"
format_stamp: "Formatted at 2026-06-19 10:28:09 on dist-test-slave-hw7q"
server_key: "077231f01a6ab758b0fc42597db0610a"
server_key_iv: "b5a478c212f18af9bf4bc5c6e5cbe290"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:09.981493 21866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:10.008580 21866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:10.011217 21866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:10.011439 21866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:10.011756 21866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:10.012163 21866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:10.012244 21866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.012337 21866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:10.012406 21866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.019703 21866 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:35155
I20260619 10:28:10.019788 21996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:35155 every 8 connection(s)
I20260619 10:28:10.020095 21866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:28:10.020539 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 21866
I20260619 10:28:10.020622 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:28:10.020817 9577 external_mini_cluster.cc:1428] Setting key 2d581bda30409d729ad66873579a4b20
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:10.030714 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:10.020813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35067 (local address 127.9.90.65:53827)
0619 10:28:10.020998 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:10.021046 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:10.021751 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:10.022080 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:10.022084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:10.022266 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:10.022460 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:10.022464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.023321 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.023322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:10.024453 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:10.024455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.024923 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.024925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:10.025070 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:10.025646 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:10.025656 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:10.027219 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:10.029124 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:10.029129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:10.029132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:10.029402 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:10.029603 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:10.029604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:10.029604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:10.029728 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:10.030036 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:10.030043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:10.030227 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 10:28:10.030332 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 10:28:10.030480 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260619 10:28:10.034077 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:10.020826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:53827 (local address 127.9.90.126:35067)
0619 10:28:10.021341 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:10.021343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:10.021352 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:28:10.021917 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:10.021918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:10.021930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:10.021987 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:10.022551 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.023219 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.024543 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.024811 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.024878 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:10.027326 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:10.027327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:10.027327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:10.027335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:10.029030 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:10.029480 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:10.029481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:10.029481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:10.029541 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:10.029812 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:10.029813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:10.029814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:10.029953 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:10.030011 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:10.033876 (+ 3865us) server_negotiation.cc:299] Negotiation successful
0619 10:28:10.033976 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":90,"threads_started":1}
I20260619 10:28:10.034735 21997 heartbeater.cc:347] Connected to a master server at 127.9.90.126:35067
I20260619 10:28:10.034840 21997 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:10.035029 21997 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
I20260619 10:28:10.035575 21785 ts_manager.cc:194] Registered new tserver with Master: 5468e3583c4249db8cbc34fc867452c0 (127.9.90.65:35155)
I20260619 10:28:10.036470 21785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:53827
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:28:10.057305 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:47015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:35067
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 10:28:10.177376 22003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:10.177641 22003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:10.177680 22003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:10.179412 22003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:10.179473 22003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:10.179513 22003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:28:10.181617 22003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.90.84:47015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:35067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:10.181859 22003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:10.182101 22003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:10.186957 22013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:10.186954 22010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:10.187013 22011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:10.187381 22003 server_base.cc:1034] running on GCE node
I20260619 10:28:10.187582 22003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:10.187816 22003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:10.188958 22003 hybrid_clock.cc:648] HybridClock initialized: now 1781864890188940 us; error 34 us; skew 500 ppm
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:10.191918 22003 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:28:10.192390 22003 webserver.cc:460] Webserver started at http://127.9.90.66:40193/ using document root <none> and password file <none>
I20260619 10:28:10.192632 22003 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:10.192684 22003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:10.192801 22003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:10.221277 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "3c8a11c6594d478d9c94874e6d24f294"
format_stamp: "Formatted at 2026-06-19 10:28:10 on dist-test-slave-hw7q"
server_key: "945f95bdc63727b1c4236b7130e30105"
server_key_iv: "b627e52c7825e7b1feab00dd9a879db0"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:10.221697 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "3c8a11c6594d478d9c94874e6d24f294"
format_stamp: "Formatted at 2026-06-19 10:28:10 on dist-test-slave-hw7q"
server_key: "945f95bdc63727b1c4236b7130e30105"
server_key_iv: "b627e52c7825e7b1feab00dd9a879db0"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:10.225306 22003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260619 10:28:10.245934 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.246131 22003 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.001s
I20260619 10:28:10.246225 22003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "3c8a11c6594d478d9c94874e6d24f294"
format_stamp: "Formatted at 2026-06-19 10:28:10 on dist-test-slave-hw7q"
server_key: "945f95bdc63727b1c4236b7130e30105"
server_key_iv: "b627e52c7825e7b1feab00dd9a879db0"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:10.246326 22003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:10.263352 22003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:10.265640 22003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:10.265861 22003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:10.266155 22003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:10.266564 22003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:10.266650 22003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.266721 22003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:10.266777 22003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.272737 22003 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:45241
I20260619 10:28:10.273195 22003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:28:10.278716 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 22003
I20260619 10:28:10.278805 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:28:10.278995 9577 external_mini_cluster.cc:1428] Setting key be75bf97ec1d0d9bee09415b1ac92b2f
I20260619 10:28:10.275602 22134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:45241 every 8 connection(s)
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:10.287015 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:10.276219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:47913 (local address 127.9.90.126:35067)
0619 10:28:10.276268 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:10.276270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:10.276509 (+ 239us) server_negotiation.cc:407] Connection header received
0619 10:28:10.277210 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:10.277211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:10.277223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:10.277252 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:10.278098 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.278887 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.280559 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.280878 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.280923 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:10.283571 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:10.283573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:10.283573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:10.283581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:10.285218 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:10.285782 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:10.285802 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:10.285802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:10.285866 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:10.286166 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:10.286167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:10.286167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:10.286304 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:10.286361 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:10.286809 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 10:28:10.286904 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:28:10.287768 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:10.276217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35067 (local address 127.9.90.66:47913)
0619 10:28:10.276373 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:10.276423 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:10.277084 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:10.277436 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:10.277440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:10.277674 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:10.277990 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:10.277996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.279015 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.279016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:10.280446 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:10.280449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.281037 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.281038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:10.281202 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:10.281776 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:10.281810 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:10.283451 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:10.285331 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:10.285338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:10.285341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:10.285688 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:10.285938 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:10.285939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:10.285939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:10.286088 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:10.286428 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:10.286434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:10.286617 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 10:28:10.287352 (+ 735us) client_negotiation.cc:240] Negotiation successful
0619 10:28:10.287521 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260619 10:28:10.288460 22135 heartbeater.cc:347] Connected to a master server at 127.9.90.126:35067
I20260619 10:28:10.288553 22135 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:10.288757 22135 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
I20260619 10:28:10.289184 21785 ts_manager.cc:194] Registered new tserver with Master: 3c8a11c6594d478d9c94874e6d24f294 (127.9.90.66:45241)
I20260619 10:28:10.289664 21785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:47913
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:28:10.326421 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:47015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:35067
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:10.445554 22140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:10.445853 22140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:10.445899 22140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:10.447670 22140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:10.447731 22140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:10.447770 22140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:28:10.450106 22140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.90.84:47015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:35067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:10.450343 22140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:10.450714 22140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:10.457275 22148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:10.458686 22145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:10.459401 22146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:10.459977 22140 server_base.cc:1034] running on GCE node
I20260619 10:28:10.460285 22140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:10.460536 22140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:10.461702 22140 hybrid_clock.cc:648] HybridClock initialized: now 1781864890461685 us; error 40 us; skew 500 ppm
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:10.464910 22140 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:28:10.465438 22140 webserver.cc:460] Webserver started at http://127.9.90.67:41849/ using document root <none> and password file <none>
I20260619 10:28:10.465703 22140 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:10.465837 22140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:10.465981 22140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:10.489099 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "799e2368941e4e65b708c5d389e34d50"
format_stamp: "Formatted at 2026-06-19 10:28:10 on dist-test-slave-hw7q"
server_key: "8965bff6cb7868f281ca3bc6bab4d76e"
server_key_iv: "6172843f5a6cbf721475d54b1916b7f2"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:10.489693 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "799e2368941e4e65b708c5d389e34d50"
format_stamp: "Formatted at 2026-06-19 10:28:10 on dist-test-slave-hw7q"
server_key: "8965bff6cb7868f281ca3bc6bab4d76e"
server_key_iv: "6172843f5a6cbf721475d54b1916b7f2"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:10.493108 22140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260619 10:28:10.512987 22155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.513293 22140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 10:28:10.513424 22140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "799e2368941e4e65b708c5d389e34d50"
format_stamp: "Formatted at 2026-06-19 10:28:10 on dist-test-slave-hw7q"
server_key: "8965bff6cb7868f281ca3bc6bab4d76e"
server_key_iv: "6172843f5a6cbf721475d54b1916b7f2"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:10.513535 22140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:10.547168 22140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:10.549216 22140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:10.549347 22140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:10.549615 22140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:10.550009 22140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:10.550046 22140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.550076 22140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:10.550124 22140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.555816 22140 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:39215
I20260619 10:28:10.556234 22140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:28:10.557930 22268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:39215 every 8 connection(s)
I20260619 10:28:10.559538 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 22140
I20260619 10:28:10.559633 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:28:10.559832 9577 external_mini_cluster.cc:1428] Setting key a34f95dce15242d8abe011ec909efd44
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:10.569633 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:10.559662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:44005 (local address 127.9.90.126:35067)
0619 10:28:10.559722 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:10.559724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:10.560134 (+ 410us) server_negotiation.cc:407] Connection header received
0619 10:28:10.560828 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:10.560829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:10.560840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:10.560870 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:10.561712 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.562509 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.563976 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.564268 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.564311 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:10.566905 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:10.566906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:10.566906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:10.566914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:10.568317 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:10.568700 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:10.568701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:10.568701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:10.568763 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:10.568994 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:10.568995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:10.568995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:10.569113 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:10.569174 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:10.569460 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 10:28:10.569539 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:28:10.569945 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:10.559334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35067 (local address 127.9.90.67:44005)
0619 10:28:10.560014 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:10.560062 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:10.560706 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:10.561054 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:10.561058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:10.561318 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:10.561603 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:10.561610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.562623 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.562624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:10.563873 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:10.563876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.564425 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.564427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:10.564559 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:10.565347 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:10.565356 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:10.566804 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:10.568405 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:10.568411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:10.568414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:10.568630 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:10.568821 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:10.568822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:10.568822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:10.568934 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:10.569172 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:10.569178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:10.569327 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 10:28:10.569433 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 10:28:10.569573 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":98,"threads_started":1}
I20260619 10:28:10.570644 22269 heartbeater.cc:347] Connected to a master server at 127.9.90.126:35067
I20260619 10:28:10.570740 22269 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:10.570897 22269 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
I20260619 10:28:10.571622 21785 ts_manager.cc:194] Registered new tserver with Master: 799e2368941e4e65b708c5d389e34d50 (127.9.90.67:39215)
I20260619 10:28:10.572115 21785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:44005
I20260619 10:28:10.572868 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:28:10.573539 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 21866
I20260619 10:28:10.578631 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 22003
I20260619 10:28:10.584903 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 22140
I20260619 10:28:10.589491 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 21753
I20260619 10:28:10.594022 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:35067
--webserver_interface=127.9.90.126
--webserver_port=36381
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:35067
--ranger_config_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:47015/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.90.84:47015/kms with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:10.672173 22272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:10.672364 22272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:10.672384 22272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:10.673475 22272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:10.673512 22272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:10.673524 22272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:10.673538 22272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:10.673561 22272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:10.675062 22272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.90.84:47015/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:35067
--ranger_config_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:35067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=36381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:10.675261 22272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:10.675450 22272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:10.679857 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:10.679839 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:10.679862 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:10.680032 22272 server_base.cc:1034] running on GCE node
I20260619 10:28:10.680341 22272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:10.680615 22272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:10.681763 22272 hybrid_clock.cc:648] HybridClock initialized: now 1781864890681751 us; error 27 us; skew 500 ppm
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:10.683979 22272 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:28:10.684381 22272 webserver.cc:460] Webserver started at http://127.9.90.126:36381/ using document root <none> and password file <none>
I20260619 10:28:10.684567 22272 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:10.684618 22272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:10.726449 22272 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:10.727111 22288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.727293 22272 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:10.727386 22272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "13a64fb8c4c54a1a84aa461cae54a6f8"
format_stamp: "Formatted at 2026-06-19 10:28:09 on dist-test-slave-hw7q"
server_key: "0bbc691963bcbbb567f0a5266da12095"
server_key_iv: "1e689e000d091a496ac7befa6a1f0309"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:10.727739 22272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:10.744763 22272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:10.746443 22272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:10.746562 22272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:10.750519 22272 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:35067
I20260619 10:28:10.750571 22340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:35067 every 8 connection(s)
I20260619 10:28:10.750914 22272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:28:10.752170 22341 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:28:10.752833 22341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Bootstrap starting.
I20260619 10:28:10.759219 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 22272
I20260619 10:28:10.759567 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:47015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:35155
--local_ip_for_outbound_sockets=127.9.90.65
--tserver_master_addrs=127.9.90.126:35067
--webserver_port=35569
--webserver_interface=127.9.90.65
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.90.84:47015/kms with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:10.760820 22341 log.cc:824] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:10.762686 22341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:28:10.762928 22341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Bootstrap complete.
I20260619 10:28:10.765062 22341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } }
I20260619 10:28:10.765296 22341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a64fb8c4c54a1a84aa461cae54a6f8, State: Initialized, Role: FOLLOWER
I20260619 10:28:10.765439 22341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } }
I20260619 10:28:10.765502 22341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:28:10.765543 22341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:28:10.765587 22341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:28:10.766594 22341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } }
I20260619 10:28:10.766682 22341 leader_election.cc:302] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13a64fb8c4c54a1a84aa461cae54a6f8; no voters:
I20260619 10:28:10.766834 22341 leader_election.cc:288] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 10:28:10.766883 22345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:28:10.767021 22345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [term 2 LEADER]: Becoming Leader. State: Replica: 13a64fb8c4c54a1a84aa461cae54a6f8, State: Running, Role: LEADER
I20260619 10:28:10.767102 22341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:10.767330 22345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } }
I20260619 10:28:10.767623 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13a64fb8c4c54a1a84aa461cae54a6f8. Latest consensus state: current_term: 2 leader_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } } }
I20260619 10:28:10.767782 22347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a64fb8c4c54a1a84aa461cae54a6f8" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35067 } } }
I20260619 10:28:10.767850 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:10.768081 22347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8 [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:10.768271 22355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:28:10.768543 22341 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 10:28:10.768764 22355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:28:10.768940 22355 catalog_manager.cc:1259] Loaded cluster ID: 15181af9467c4161ab47b5137f086cfd
I20260619 10:28:10.769011 22355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:28:10.772915 22355 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:28:10.773255 22355 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Loaded TSK: 0
I20260619 10:28:10.773535 22355 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:28:10.843075 22343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:10.843257 22343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:10.843286 22343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:10.844426 22343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:10.844465 22343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:10.844498 22343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:28:10.846109 22343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.90.84:47015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:35155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=35569
--tserver_master_addrs=127.9.90.126:35067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:10.846328 22343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:10.846575 22343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:10.851289 22378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:10.851327 22381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:10.851591 22379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:10.851800 22343 server_base.cc:1034] running on GCE node
I20260619 10:28:10.851992 22343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:10.852217 22343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:10.853358 22343 hybrid_clock.cc:648] HybridClock initialized: now 1781864890853345 us; error 26 us; skew 500 ppm
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:10.856078 22343 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:28:10.856452 22343 webserver.cc:460] Webserver started at http://127.9.90.65:35569/ using document root <none> and password file <none>
I20260619 10:28:10.856647 22343 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:10.856705 22343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:10.875949 22343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:28:10.877051 22389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.877262 22343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:28:10.877351 22343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "5468e3583c4249db8cbc34fc867452c0"
format_stamp: "Formatted at 2026-06-19 10:28:09 on dist-test-slave-hw7q"
server_key: "077231f01a6ab758b0fc42597db0610a"
server_key_iv: "b5a478c212f18af9bf4bc5c6e5cbe290"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:10.877700 22343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:10.891774 22343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:10.893774 22343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:10.893934 22343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:10.894152 22343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:10.894600 22343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:10.894650 22343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.894711 22343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:10.894747 22343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:10.901250 22343 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:35155
I20260619 10:28:10.901609 22343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:28:10.907554 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 22343
I20260619 10:28:10.907827 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:47015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:45241
--local_ip_for_outbound_sockets=127.9.90.66
--tserver_master_addrs=127.9.90.126:35067
--webserver_port=40193
--webserver_interface=127.9.90.66
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.90.84:47015/kms with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:10.910635 22502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:35155 every 8 connection(s)
Jun 19 10:28:10 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:10.918608 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:10.906614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35067 (local address 127.9.90.65:57359)
0619 10:28:10.907246 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:10.907290 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:10.907917 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:10.908570 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:10.908574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:10.908813 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:10.909079 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:10.909086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.910728 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.910729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:10.911499 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:10.911501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.912347 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.912348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:10.912467 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:10.912999 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:10.913008 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:10.914589 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:10.916765 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:10.916771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:10.916774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:10.917049 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:10.917332 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:10.917333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:10.917334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:10.917461 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:10.917949 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:10.917955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:10.918122 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 10:28:10.918227 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 10:28:10.918379 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":80,"threads_started":1}
I20260619 10:28:10.919379 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:10.906973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:57359 (local address 127.9.90.126:35067)
0619 10:28:10.907193 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:10.907203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:10.908072 (+ 869us) server_negotiation.cc:407] Connection header received
0619 10:28:10.908229 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:10.908236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:10.908294 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:10.908411 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:10.909218 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.910614 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.911581 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:10.912147 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:10.912242 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:10.914730 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:10.914735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:10.914735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:10.914748 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:10.916676 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:10.917152 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:10.917155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:10.917155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:10.917261 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:10.917532 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:10.917533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:10.917533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:10.917839 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:10.917918 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:10.919016 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0619 10:28:10.919145 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260619 10:28:10.920480 22503 heartbeater.cc:347] Connected to a master server at 127.9.90.126:35067
I20260619 10:28:10.920567 22503 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:10.920722 22503 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
W20260619 10:28:10.921530 22503 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:35067 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 10:28:10.921602 22503 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:10.921666 22503 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
I20260619 10:28:10.922008 22503 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:10.922081 22503 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
W20260619 10:28:10.922405 22503 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:28:11.005054 22507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:11.005474 22507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:11.005545 22507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:11.007302 22507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:11.007411 22507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:11.007469 22507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:28:11.009675 22507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.90.84:47015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:45241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=40193
--tserver_master_addrs=127.9.90.126:35067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:11.010169 22507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:11.010474 22507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:11.016716 22513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:11.016819 22512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:11.016706 22515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:11.017575 22507 server_base.cc:1034] running on GCE node
I20260619 10:28:11.017885 22507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:11.018165 22507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:11.019332 22507 hybrid_clock.cc:648] HybridClock initialized: now 1781864891019317 us; error 27 us; skew 500 ppm
Jun 19 10:28:11 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:11.022534 22507 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:28:11.023032 22507 webserver.cc:460] Webserver started at http://127.9.90.66:40193/ using document root <none> and password file <none>
I20260619 10:28:11.023275 22507 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:11.023336 22507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 10:28:11 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:11.038440 22507 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 10:28:11.039340 22523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:11.039523 22507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:11.039602 22507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "3c8a11c6594d478d9c94874e6d24f294"
format_stamp: "Formatted at 2026-06-19 10:28:10 on dist-test-slave-hw7q"
server_key: "945f95bdc63727b1c4236b7130e30105"
server_key_iv: "b627e52c7825e7b1feab00dd9a879db0"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:11.040035 22507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:11.053660 22507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:11.055580 22507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:11.055720 22507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:11.055905 22507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:11.056229 22507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:11.056264 22507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:11.056293 22507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:11.056339 22507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:11.062683 22507 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:45241
I20260619 10:28:11.062788 22636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:45241 every 8 connection(s)
I20260619 10:28:11.063109 22507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:28:11.064792 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 22507
I20260619 10:28:11.065042 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.90.84:47015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:39215
--local_ip_for_outbound_sockets=127.9.90.67
--tserver_master_addrs=127.9.90.126:35067
--webserver_port=41849
--webserver_interface=127.9.90.67
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.90.84:47015/kms with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 10:28:11 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:11.074371 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:11.063910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35067 (local address 127.9.90.66:56351)
0619 10:28:11.064050 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:11.064091 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:11.064558 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:11.066083 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:11.066088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:11.066313 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:11.066502 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:11.066506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.067446 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.067447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:11.068279 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:11.068282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.068773 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.068774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:11.068940 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:11.069430 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:11.069440 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:11.070765 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:11.072647 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:11.072653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:11.072656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:11.072939 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:11.073162 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:11.073163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:11.073163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:11.073349 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:11.073642 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:11.073648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:11.073852 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 10:28:11.073965 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 10:28:11.074135 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260619 10:28:11.074744 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:11.063906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:56351 (local address 127.9.90.126:35067)
0619 10:28:11.063951 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:11.063954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:11.064168 (+ 214us) server_negotiation.cc:407] Connection header received
0619 10:28:11.065866 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:11.065867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:11.065884 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:11.065942 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:11.066590 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.067366 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.068355 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.068633 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.068694 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:11.070873 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:11.070875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:11.070875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:11.070883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:11.072554 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:11.073022 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:11.073023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:11.073023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:11.073082 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:11.073420 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:11.073421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:11.073422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:11.073559 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:11.073616 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:11.074566 (+ 950us) server_negotiation.cc:299] Negotiation successful
0619 10:28:11.074654 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:28:11.075369 22637 heartbeater.cc:347] Connected to a master server at 127.9.90.126:35067
I20260619 10:28:11.075488 22637 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:11.075696 22637 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
W20260619 10:28:11.076423 22637 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:35067 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 10:28:11.076504 22637 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:11.076583 22637 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
I20260619 10:28:11.076930 22637 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:11.077011 22637 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
W20260619 10:28:11.077322 22637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:28:11.184176 22640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:11.184463 22640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:11.184507 22640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:11.186257 22640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:11.186314 22640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:11.186353 22640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:28:11.188511 22640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.90.84:47015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:39215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=41849
--tserver_master_addrs=127.9.90.126:35067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:11.188771 22640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:11.189095 22640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:11.195000 22645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:11.195433 22646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:11.195719 22640 server_base.cc:1034] running on GCE node
W20260619 10:28:11.195789 22648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:11.196074 22640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:11.196388 22640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:11.197577 22640 hybrid_clock.cc:648] HybridClock initialized: now 1781864891197550 us; error 42 us; skew 500 ppm
Jun 19 10:28:11 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:11.200807 22640 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:28:11.201309 22640 webserver.cc:460] Webserver started at http://127.9.90.67:41849/ using document root <none> and password file <none>
I20260619 10:28:11.201558 22640 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:11.201622 22640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 10:28:11 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:11.225006 22640 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260619 10:28:11.226011 22656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:11.226198 22640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:28:11.226280 22640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "799e2368941e4e65b708c5d389e34d50"
format_stamp: "Formatted at 2026-06-19 10:28:10 on dist-test-slave-hw7q"
server_key: "8965bff6cb7868f281ca3bc6bab4d76e"
server_key_iv: "6172843f5a6cbf721475d54b1916b7f2"
server_key_version: "kuduclusterkey@0"
I20260619 10:28:11.226722 22640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:11.273177 22640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:11.279102 22640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:11.279314 22640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:11.279613 22640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:11.280076 22640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:11.280144 22640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:11.280189 22640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:11.280220 22640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:11.286398 22640 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:39215
I20260619 10:28:11.286875 22640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:28:11.289999 22770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:39215 every 8 connection(s)
I20260619 10:28:11.293988 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 22640
Jun 19 10:28:11 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:11.301596 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:11.290991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:35411 (local address 127.9.90.126:35067)
0619 10:28:11.291036 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:11.291038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:11.291874 (+ 836us) server_negotiation.cc:407] Connection header received
0619 10:28:11.292569 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:11.292571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:11.292585 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:11.292616 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:11.293423 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.294632 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.296109 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.296329 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.296387 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:11.298609 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:11.298610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:11.298611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:11.298617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:11.300058 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:11.300504 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:11.300506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:11.300506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:11.300559 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:11.300838 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:11.300839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:11.300839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:11.300967 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:11.301007 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:11.301390 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 10:28:11.301498 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:28:11.302093 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:11.290710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:35067 (local address 127.9.90.67:35411)
0619 10:28:11.291730 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:11.291779 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:11.292435 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:11.292812 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:11.292817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:11.293025 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:11.293315 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:11.293322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.294746 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.294747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:11.296017 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:11.296020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.296424 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.296426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:11.296540 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:11.296994 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:11.297002 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:11.298522 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:11.300130 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:11.300137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:11.300139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:11.300418 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:11.300629 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:11.300630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:11.300630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:11.300764 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:11.301054 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:11.301060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:11.301237 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 10:28:11.301712 (+ 475us) client_negotiation.cc:240] Negotiation successful
0619 10:28:11.301881 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":77,"threads_started":1}
I20260619 10:28:11.302691 22771 heartbeater.cc:347] Connected to a master server at 127.9.90.126:35067
I20260619 10:28:11.302790 22771 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:11.303707 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:11.294575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.9.90.126:35067)
0619 10:28:11.295502 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:11.295504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:11.295509 (+ 5us) server_negotiation.cc:407] Connection header received
0619 10:28:11.295525 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:11.295526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:11.295534 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:11.295591 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:11.296415 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.297163 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.298065 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:11.298272 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:11.298322 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:11.300033 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:11.300034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:11.300035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:11.300042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:11.301605 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:11.301997 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:11.301998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:11.301999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:11.302063 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:11.302761 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:11.302762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:11.302762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:11.302993 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:11.303166 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:11.303555 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 10:28:11.303644 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":53,"threads_started":1}
I20260619 10:28:11.304134 22771 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
W20260619 10:28:11.304813 22771 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:35067 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 10:28:11.304888 22771 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:11.304945 22771 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
I20260619 10:28:11.305333 22771 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:11.305434 22771 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
W20260619 10:28:11.305807 22771 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 10:28:11 dist-test-slave-hw7q krb5kdc[21309](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781864891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:11.922559 22503 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:11.922719 22503 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
I20260619 10:28:11.923033 22305 ts_manager.cc:194] Registered new tserver with Master: 5468e3583c4249db8cbc34fc867452c0 (127.9.90.65:35155)
I20260619 10:28:11.923692 22305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:57359
I20260619 10:28:12.077514 22637 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:12.077684 22637 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
I20260619 10:28:12.078047 22305 ts_manager.cc:194] Registered new tserver with Master: 3c8a11c6594d478d9c94874e6d24f294 (127.9.90.66:45241)
I20260619 10:28:12.078622 22305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:56351
Jun 19 10:28:12 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:12.305969 22771 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:12.306111 22771 heartbeater.cc:511] Master 127.9.90.126:35067 requested a full tablet report, sending...
I20260619 10:28:12.306350 22305 ts_manager.cc:194] Registered new tserver with Master: 799e2368941e4e65b708c5d389e34d50 (127.9.90.67:39215)
I20260619 10:28:12.306730 22305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:35411
I20260619 10:28:12.313184 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:28:12.324929 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.314142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.9.90.126:35067)
0619 10:28:12.314267 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:12.314269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:12.317807 (+ 3538us) server_negotiation.cc:407] Connection header received
0619 10:28:12.317864 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:12.317865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:12.317879 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:12.317954 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:12.318604 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.319328 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.320353 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.320639 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.320692 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.322111 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:12.322112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:12.322113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:12.322120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:12.323722 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:12.324102 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:12.324103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:12.324103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:12.324157 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:12.324347 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:12.324347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:12.324348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:12.324462 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:12.324543 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:12.324675 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 10:28:12.324748 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260619 10:28:12.326957 22305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:28:12.329288 22305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:28:12.338235 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.332189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:35155 (local address 127.0.0.1:57278)
0619 10:28:12.333943 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:12.333952 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:12.334005 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:12.334276 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:12.334279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:12.334281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:12.334348 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.334350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.335576 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.335577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.336872 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.336876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.337941 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.337942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.338034 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.338050 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 10:28:12.338111 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:28:12.338158 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1738,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260619 10:28:12.338384 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.332517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.9.90.66:45241)
0619 10:28:12.334096 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:12.334099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:12.334108 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:28:12.334139 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:12.334141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:12.334155 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:12.334218 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:12.334409 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.335687 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.336956 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.337946 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.338004 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.338055 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:12.338272 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 10:28:12.338310 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1548,"thread_start_us":48,"threads_started":1}
I20260619 10:28:12.338557 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.332151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45241 (local address 127.0.0.1:47284)
0619 10:28:12.332627 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:12.332675 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:12.332738 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:12.334295 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:12.334296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:12.334298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:12.334349 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.334351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.335766 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.335767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.336870 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.336873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.338087 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.338088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.338147 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.338152 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:28:12.338196 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 10:28:12.338233 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":9,"thread_start_us":65,"threads_started":1}
I20260619 10:28:12.338915 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.332391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.9.90.65:35155)
0619 10:28:12.332566 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:12.332571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:12.334025 (+ 1454us) server_negotiation.cc:407] Connection header received
0619 10:28:12.334069 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:12.334072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:12.334090 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:12.334177 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:12.334425 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.335485 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.336978 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.337738 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.337840 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.337909 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:12.338794 (+ 885us) server_negotiation.cc:299] Negotiation successful
0619 10:28:12.338838 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260619 10:28:12.339428 22571 tablet_service.cc:1467] Processing CreateTablet for tablet eedf8fd13df04357befa0fc6315cf0c2 (DEFAULT_TABLE table=test-table [id=ca6221de228148e5b6323325abf8f3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:12.339741 22571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedf8fd13df04357befa0fc6315cf0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:12.339689 22437 tablet_service.cc:1467] Processing CreateTablet for tablet eedf8fd13df04357befa0fc6315cf0c2 (DEFAULT_TABLE table=test-table [id=ca6221de228148e5b6323325abf8f3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:12.340029 22437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedf8fd13df04357befa0fc6315cf0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:12.344378 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.332435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.9.90.67:39215)
0619 10:28:12.333887 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:12.333892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:12.337912 (+ 4020us) server_negotiation.cc:407] Connection header received
0619 10:28:12.338026 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:12.338029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:12.338044 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:12.338095 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:12.338310 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.342185 (+ 3875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.343237 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.343982 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.344045 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.344096 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:12.344265 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 10:28:12.344305 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1419,"thread_start_us":46,"threads_started":1}
I20260619 10:28:12.344743 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.332342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:39215 (local address 127.0.0.1:46040)
0619 10:28:12.337845 (+ 5503us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:12.337855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:12.337941 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:12.338195 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:12.338195 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:12.338197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:12.338239 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.338240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.342281 (+ 4041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.342282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.343160 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.343163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.344129 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.344130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.344187 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.344191 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:28:12.344634 (+ 443us) client_negotiation.cc:240] Negotiation successful
0619 10:28:12.344677 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5486,"thread_start_us":58,"threads_started":1}
I20260619 10:28:12.345541 22705 tablet_service.cc:1467] Processing CreateTablet for tablet eedf8fd13df04357befa0fc6315cf0c2 (DEFAULT_TABLE table=test-table [id=ca6221de228148e5b6323325abf8f3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:12.345899 22705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedf8fd13df04357befa0fc6315cf0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:12.345932 22805 tablet_bootstrap.cc:492] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0: Bootstrap starting.
I20260619 10:28:12.346594 22805 tablet_bootstrap.cc:654] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:12.346917 22805 log.cc:824] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:12.347252 22804 tablet_bootstrap.cc:492] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: Bootstrap starting.
I20260619 10:28:12.347867 22804 tablet_bootstrap.cc:654] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:12.348203 22804 log.cc:824] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:12.349278 22808 tablet_bootstrap.cc:492] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: Bootstrap starting.
I20260619 10:28:12.349908 22808 tablet_bootstrap.cc:654] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:12.350181 22808 log.cc:824] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:12.354483 22804 tablet_bootstrap.cc:492] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: No bootstrap required, opened a new log
I20260619 10:28:12.354560 22804 ts_tablet_manager.cc:1397] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 10:28:12.354563 22808 tablet_bootstrap.cc:492] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: No bootstrap required, opened a new log
I20260619 10:28:12.354640 22808 ts_tablet_manager.cc:1397] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 10:28:12.357036 22808 raft_consensus.cc:348] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } }
I20260619 10:28:12.357193 22808 raft_consensus.cc:374] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:12.357082 22804 raft_consensus.cc:348] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } }
I20260619 10:28:12.357224 22808 raft_consensus.cc:729] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 799e2368941e4e65b708c5d389e34d50, State: Initialized, Role: FOLLOWER
I20260619 10:28:12.357242 22804 raft_consensus.cc:374] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:12.357275 22804 raft_consensus.cc:729] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8a11c6594d478d9c94874e6d24f294, State: Initialized, Role: FOLLOWER
I20260619 10:28:12.357353 22808 consensus_queue.cc:260] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } }
I20260619 10:28:12.357596 22808 ts_tablet_manager.cc:1428] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:28:12.357699 22771 heartbeater.cc:503] Master 127.9.90.126:35067 was elected leader, sending a full tablet report...
I20260619 10:28:12.357702 22805 tablet_bootstrap.cc:492] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0: No bootstrap required, opened a new log
I20260619 10:28:12.357762 22805 ts_tablet_manager.cc:1397] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260619 10:28:12.358273 22804 consensus_queue.cc:260] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } }
I20260619 10:28:12.358547 22804 ts_tablet_manager.cc:1428] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 10:28:12.358685 22637 heartbeater.cc:503] Master 127.9.90.126:35067 was elected leader, sending a full tablet report...
I20260619 10:28:12.360040 22805 raft_consensus.cc:348] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } }
I20260619 10:28:12.360203 22805 raft_consensus.cc:374] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:12.360242 22805 raft_consensus.cc:729] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5468e3583c4249db8cbc34fc867452c0, State: Initialized, Role: FOLLOWER
I20260619 10:28:12.360347 22805 consensus_queue.cc:260] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } }
I20260619 10:28:12.360599 22805 ts_tablet_manager.cc:1428] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:28:12.360729 22503 heartbeater.cc:503] Master 127.9.90.126:35067 was elected leader, sending a full tablet report...
W20260619 10:28:12.408161 22504 tablet.cc:2369] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:28:12.439487 22810 raft_consensus.cc:482] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:12.439646 22810 raft_consensus.cc:504] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } }
I20260619 10:28:12.440122 22810 leader_election.cc:288] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c8a11c6594d478d9c94874e6d24f294 (127.9.90.66:45241), 5468e3583c4249db8cbc34fc867452c0 (127.9.90.65:35155)
I20260619 10:28:12.444357 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.440587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:57435 (local address 127.9.90.66:45241)
0619 10:28:12.440678 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:12.440680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:12.440689 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:28:12.440716 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:12.440717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:12.440728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:12.440787 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:12.440990 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.441755 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.443297 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.443918 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.443965 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.443976 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:12.444239 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 10:28:12.444282 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 10:28:12.444614 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.440291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45241 (local address 127.9.90.67:57435)
0619 10:28:12.440523 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:12.440534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:12.440602 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:12.440839 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:12.440840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:12.440843 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:12.440919 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.440921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.441896 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.441897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.443190 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.443193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.444088 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.444089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.444161 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.444165 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:28:12.444481 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 10:28:12.444523 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260619 10:28:12.444964 22591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedf8fd13df04357befa0fc6315cf0c2" candidate_uuid: "799e2368941e4e65b708c5d389e34d50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8a11c6594d478d9c94874e6d24f294" is_pre_election: true
I20260619 10:28:12.445170 22591 raft_consensus.cc:2454] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 799e2368941e4e65b708c5d389e34d50 in term 0.
I20260619 10:28:12.445438 22660 leader_election.cc:302] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8a11c6594d478d9c94874e6d24f294, 799e2368941e4e65b708c5d389e34d50; no voters:
I20260619 10:28:12.445576 22810 raft_consensus.cc:2781] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:28:12.445621 22810 raft_consensus.cc:482] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:28:12.445643 22810 raft_consensus.cc:3037] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:12.445741 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.440633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:45483 (local address 127.9.90.65:35155)
0619 10:28:12.440701 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:12.440703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:12.441596 (+ 893us) server_negotiation.cc:407] Connection header received
0619 10:28:12.441690 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:12.441691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:12.441704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:12.441730 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:12.441959 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.442594 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.443880 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.444422 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.444470 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.444480 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:12.445631 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0619 10:28:12.445677 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 10:28:12.445989 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.440331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:35155 (local address 127.9.90.67:45483)
0619 10:28:12.441515 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:12.441526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:12.441623 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:12.441810 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:12.441811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:12.441812 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:12.441883 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.441885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.442698 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.442699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.443782 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:12.443785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.445468 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.445468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:12.445537 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.445553 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 10:28:12.445865 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 10:28:12.445908 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":78,"threads_started":1}
I20260619 10:28:12.446375 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedf8fd13df04357befa0fc6315cf0c2" candidate_uuid: "799e2368941e4e65b708c5d389e34d50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5468e3583c4249db8cbc34fc867452c0" is_pre_election: true
I20260619 10:28:12.446518 22447 raft_consensus.cc:2454] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 799e2368941e4e65b708c5d389e34d50 in term 0.
I20260619 10:28:12.446870 22810 raft_consensus.cc:504] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } }
I20260619 10:28:12.447010 22810 leader_election.cc:288] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [CANDIDATE]: Term 1 election: Requested vote from peers 3c8a11c6594d478d9c94874e6d24f294 (127.9.90.66:45241), 5468e3583c4249db8cbc34fc867452c0 (127.9.90.65:35155)
I20260619 10:28:12.447168 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedf8fd13df04357befa0fc6315cf0c2" candidate_uuid: "799e2368941e4e65b708c5d389e34d50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5468e3583c4249db8cbc34fc867452c0"
I20260619 10:28:12.447219 22591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedf8fd13df04357befa0fc6315cf0c2" candidate_uuid: "799e2368941e4e65b708c5d389e34d50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8a11c6594d478d9c94874e6d24f294"
I20260619 10:28:12.447243 22447 raft_consensus.cc:3037] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:12.447290 22591 raft_consensus.cc:3037] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:12.448061 22447 raft_consensus.cc:2454] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 799e2368941e4e65b708c5d389e34d50 in term 1.
I20260619 10:28:12.448228 22657 leader_election.cc:302] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5468e3583c4249db8cbc34fc867452c0, 799e2368941e4e65b708c5d389e34d50; no voters:
I20260619 10:28:12.448313 22810 raft_consensus.cc:2781] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:28:12.448453 22810 raft_consensus.cc:686] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 1 LEADER]: Becoming Leader. State: Replica: 799e2368941e4e65b708c5d389e34d50, State: Running, Role: LEADER
I20260619 10:28:12.448450 22591 raft_consensus.cc:2454] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 799e2368941e4e65b708c5d389e34d50 in term 1.
I20260619 10:28:12.448550 22810 consensus_queue.cc:237] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } }
I20260619 10:28:12.449602 22304 catalog_manager.cc:5495] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 799e2368941e4e65b708c5d389e34d50 (127.9.90.67). New cstate: current_term: 1 leader_uuid: "799e2368941e4e65b708c5d389e34d50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "799e2368941e4e65b708c5d389e34d50" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 39215 } health_report { overall_health: HEALTHY } } }
Jun 19 10:28:12 dist-test-slave-hw7q krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for HTTP/127.9.90.84@KRBTEST.COM
I20260619 10:28:12.465297 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.461962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.9.90.67:39215)
0619 10:28:12.462013 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:12.462015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:12.462024 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:28:12.462074 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:12.462075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:12.462099 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:12.462188 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:28:12.462350 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.463084 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.464179 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:12.464482 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:12.464540 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:12.464754 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:28:12.464894 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:28:12.464966 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:12.465195 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 10:28:12.465215 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:28:12.467687 22591 raft_consensus.cc:1261] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 1 FOLLOWER]: Refusing update from remote peer 799e2368941e4e65b708c5d389e34d50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:12.468097 22447 raft_consensus.cc:1261] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [term 1 FOLLOWER]: Refusing update from remote peer 799e2368941e4e65b708c5d389e34d50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:12.468295 22816 consensus_queue.cc:1035] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8a11c6594d478d9c94874e6d24f294" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 45241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:12.468652 22816 consensus_queue.cc:1035] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5468e3583c4249db8cbc34fc867452c0" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 35155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:12.472692 22819 mvcc.cc:205] Tried to move back new op lower bound from 7298518599543963648 to 7298518599469834240. Current Snapshot: MvccSnapshot[applied={T|T < 7298518599543963648}]
I20260619 10:28:12.476506 22304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:12.476596 22304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:12.477905 22304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13a64fb8c4c54a1a84aa461cae54a6f8: Sending DeleteTablet for 3 replicas of tablet eedf8fd13df04357befa0fc6315cf0c2
I20260619 10:28:12.478335 22571 tablet_service.cc:1514] Processing DeleteTablet for tablet eedf8fd13df04357befa0fc6315cf0c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:12 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:47284
I20260619 10:28:12.478920 22705 tablet_service.cc:1514] Processing DeleteTablet for tablet eedf8fd13df04357befa0fc6315cf0c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:12 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46040
I20260619 10:28:12.479017 22437 tablet_service.cc:1514] Processing DeleteTablet for tablet eedf8fd13df04357befa0fc6315cf0c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:12 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:57278
I20260619 10:28:12.479336 22829 tablet_replica.cc:330] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0: stopping tablet replica
I20260619 10:28:12.479465 22829 raft_consensus.cc:2229] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:28:12.479525 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 22343
I20260619 10:28:12.479645 22829 raft_consensus.cc:2258] T eedf8fd13df04357befa0fc6315cf0c2 P 5468e3583c4249db8cbc34fc867452c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:12.481916 22827 tablet_replica.cc:330] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: stopping tablet replica
I20260619 10:28:12.482106 22827 raft_consensus.cc:2229] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:28:12.482234 22828 tablet_replica.cc:330] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: stopping tablet replica
I20260619 10:28:12.482344 22827 pending_rounds.cc:62] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: Trying to abort 2 pending ops.
I20260619 10:28:12.482419 22827 pending_rounds.cc:68] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298518599543963648 op_type: WRITE_OP write_request { tablet_id: "eedf8fd13df04357befa0fc6315cf0c2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\350\265\324\321\006\"8\n\ntest-admin\022*\n ca6221de228148e5b6323325abf8f3d5\020\001\030\001 \001(\001" signature: "<redacted>""\226D\310y\220|\004\327\340\205aN\260\366\214\031\273\274\315\206\372\276\375\315\205S\361\352\306\245\324kSh\001f\226\270\220a;\352Y\275fK\305\357@\357\017\037<\353\317s\307\275\242 \206H\256\365" signing_key_seq_num: 0 } } request_id { client_id: "4b6b87e39ff046f8a069596209ea1dff" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260619 10:28:12.482685 22828 raft_consensus.cc:2229] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:28:12.482856 22828 raft_consensus.cc:2258] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:12.483062 22827 pending_rounds.cc:68] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298518599469834240 op_type: NO_OP noop_request { }
I20260619 10:28:12.483575 22828 ts_tablet_manager.cc:1905] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:12.483669 22827 raft_consensus.cc:2866] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260619 10:28:12.483753 22827 raft_consensus.cc:2258] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:12.484195 22827 ts_tablet_manager.cc:1905] T eedf8fd13df04357befa0fc6315cf0c2 P 3c8a11c6594d478d9c94874e6d24f294: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 10:28:12.486245 22289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:35155 (error 108)
I20260619 10:28:12.486822 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 22507
I20260619 10:28:12.489907 22828 ts_tablet_manager.cc:1918] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:12.489985 22828 log.cc:1196] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/eedf8fd13df04357befa0fc6315cf0c2
I20260619 10:28:12.490171 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:12.490008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:35155 (local address 127.0.0.1:57292)
0619 10:28:12.490097 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:12.490137 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:35155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 10:28:12.490257 22828 ts_tablet_manager.cc:1939] T eedf8fd13df04357befa0fc6315cf0c2 P 799e2368941e4e65b708c5d389e34d50: Deleting consensus metadata
W20260619 10:28:12.490303 22289 catalog_manager.cc:4584] TS 5468e3583c4249db8cbc34fc867452c0 (127.9.90.65:35155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eedf8fd13df04357befa0fc6315cf0c2: Network error: Client connection negotiation failed: client connection to 127.9.90.65:35155: connect: Connection refused (error 111)
I20260619 10:28:12.493630 22292 catalog_manager.cc:4841] TS 799e2368941e4e65b708c5d389e34d50 (127.9.90.67:39215): tablet eedf8fd13df04357befa0fc6315cf0c2 (table test-table [id=ca6221de228148e5b6323325abf8f3d5]) successfully deleted
W20260619 10:28:12.494773 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:28:12.494373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:45241 (local address 127.0.0.1:47294)
0619 10:28:12.494412 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:12.494422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:12.494456 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:12.494730 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.66:45241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 10:28:12.495160 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 22640
I20260619 10:28:12.504976 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 22272
I20260619 10:28:12.511363 9577 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 10:28:12.855595 9577 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 10:28:12.855679 9577 mini_ranger.cc:66] Stopping Ranger...
I20260619 10:28:13.189760 9577 mini_ranger.cc:68] Stopped Ranger
2026-06-19 10:28:13.189 UTC [21339] LOG: received smart shutdown request
2026-06-19 10:28:13.193 UTC [21339] LOG: background worker "logical replication launcher" (PID 21348) exited with exit code 1
2026-06-19 10:28:13.193 UTC [21343] LOG: shutting down
2026-06-19 10:28:13.329 UTC [21339] LOG: database system is shut down
2026-06-19T10:28:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44334 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:28:13 dist-test-slave-hw7q krb5kdc[22840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:28:13 dist-test-slave-hw7q krb5kdc[22840](info): set up 2 sockets
Jun 19 10:28:13 dist-test-slave-hw7q krb5kdc[22840](info): commencing operation
krb5kdc: starting...
W20260619 10:28:14.357455 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:28:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:28:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:28:14.468251 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42145
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:42145,127.9.90.125:44353,127.9.90.124:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:14.545241 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:14.545433 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:14.545454 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:14.546651 22853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:14.546707 22853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:14.546721 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:14.546733 22853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:14.546744 22853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:14.548633 22853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:42145,127.9.90.125:44353,127.9.90.124:46361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:14.548840 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:14.549022 22853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:14.553606 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:14.553583 22861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:14.553627 22853 server_base.cc:1034] running on GCE node
W20260619 10:28:14.553593 22859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:14.554085 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:14.554335 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:14.555480 22853 hybrid_clock.cc:648] HybridClock initialized: now 1781864894555466 us; error 33 us; skew 500 ppm
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:14.557737 22853 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:28:14.558190 22853 webserver.cc:460] Webserver started at http://127.9.90.126:40633/ using document root <none> and password file <none>
I20260619 10:28:14.558388 22853 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:14.558434 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:14.558555 22853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:14.559537 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "407c06e76b20455cb8c1b042616eaaa1"
format_stamp: "Formatted at 2026-06-19 10:28:14 on dist-test-slave-hw7q"
server_key: "f84aa3ed9030df7032bf48af90bd2411"
server_key_iv: "73da65dd7cee5d4469cdbf89e0c000c9"
server_key_version: "encryptionkey@0"
I20260619 10:28:14.559839 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "407c06e76b20455cb8c1b042616eaaa1"
format_stamp: "Formatted at 2026-06-19 10:28:14 on dist-test-slave-hw7q"
server_key: "f84aa3ed9030df7032bf48af90bd2411"
server_key_iv: "73da65dd7cee5d4469cdbf89e0c000c9"
server_key_version: "encryptionkey@0"
I20260619 10:28:14.560992 22853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:28:14.561652 22868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:14.561829 22853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:14.561911 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "407c06e76b20455cb8c1b042616eaaa1"
format_stamp: "Formatted at 2026-06-19 10:28:14 on dist-test-slave-hw7q"
server_key: "f84aa3ed9030df7032bf48af90bd2411"
server_key_iv: "73da65dd7cee5d4469cdbf89e0c000c9"
server_key_version: "encryptionkey@0"
I20260619 10:28:14.561986 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:14.570673 22853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:14.572667 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:14.572798 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:14.576512 22853 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:42145
I20260619 10:28:14.576542 22920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:42145 every 8 connection(s)
I20260619 10:28:14.576858 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:28:14.577664 22921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:14.579165 22921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:28:14.582444 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 22853
I20260619 10:28:14.582522 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:28:14.582669 9577 external_mini_cluster.cc:1428] Setting key d26089c7ba1af55a18956285ba970e3b
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:14.589102 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.579849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.9.90.126:42145)
0619 10:28:14.580027 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.580035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.580147 (+ 112us) server_negotiation.cc:407] Connection header received
0619 10:28:14.580678 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.580680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.580697 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.580780 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.581303 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.582124 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.583068 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.583516 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.583627 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.585923 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.585930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.585930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.585936 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.587481 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.587946 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.587948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.587948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.588029 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.588262 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.588263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.588263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.588457 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.588514 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.588762 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.588928 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260619 10:28:14.589107 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.579836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42145 (local address 127.0.0.1:58806)
0619 10:28:14.580035 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.580075 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:14.580545 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:14.580795 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:14.580797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:14.581060 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:14.581233 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.581237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.582218 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.582219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.582993 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.582996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.583613 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.583614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.583692 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.584195 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:14.584203 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:14.585824 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:14.587567 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.587574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.587577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.587835 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.588095 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.588096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.588096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.588204 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.588519 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:14.588524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:14.588608 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 10:28:14.588710 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 10:28:14.588876 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.9.90.125@KRBTEST.COM; defaulting to no policy
I20260619 10:28:14.589937 22921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:28:14.590267 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.590159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:44353 (local address 127.0.0.1:50172)
0619 10:28:14.590200 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.590243 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 10:28:14.590440 22872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
I20260619 10:28:14.590803 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.590728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:44353 (local address 127.0.0.1:50188)
0619 10:28:14.590764 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.590788 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 10:28:14.590890 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.125:44353: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab.
Entry for principal kudu/127.9.90.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab.
I20260619 10:28:14.606254 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.125:44353
--webserver_interface=127.9.90.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab
--principal=kudu/127.9.90.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:42145,127.9.90.125:44353,127.9.90.124:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:14.640129 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } attempt: 1
I20260619 10:28:14.640722 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.640553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:44353 (local address 127.0.0.1:50204)
0619 10:28:14.640625 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.640680 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 10:28:14.641142 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.641041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:44353 (local address 127.0.0.1:50216)
0619 10:28:14.641083 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.641120 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:28:14.641263 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.125:44353: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
W20260619 10:28:14.684156 22928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:14.684362 22928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:14.684383 22928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:14.685491 22928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:14.685532 22928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:14.685545 22928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:14.685556 22928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:14.685567 22928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:14.687043 22928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:42145,127.9.90.125:44353,127.9.90.124:46361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab
--principal=kudu/127.9.90.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.125:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.90.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:14.687258 22928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:14.687464 22928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:14.692003 22934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:14.692021 22928 server_base.cc:1034] running on GCE node
W20260619 10:28:14.692027 22933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:14.692003 22936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:14.692399 22928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:14.692678 22928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:14.693833 22928 hybrid_clock.cc:648] HybridClock initialized: now 1781864894693819 us; error 28 us; skew 500 ppm
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:14.696193 22928 init.cc:377] Logged in from keytab as kudu/127.9.90.125@KRBTEST.COM (short username kudu)
I20260619 10:28:14.696631 22928 webserver.cc:460] Webserver started at http://127.9.90.125:43195/ using document root <none> and password file <none>
I20260619 10:28:14.696830 22928 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:14.696877 22928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:14.696998 22928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:14.698105 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/data/instance:
uuid: "0ed2fede092140e0a4996cd75be9a850"
format_stamp: "Formatted at 2026-06-19 10:28:14 on dist-test-slave-hw7q"
server_key: "c0bf9ad747187d21a843f6dd901f8779"
server_key_iv: "2ecad20aa1feb30b78e808e9b60d18a4"
server_key_version: "encryptionkey@0"
I20260619 10:28:14.698513 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/wal/instance:
uuid: "0ed2fede092140e0a4996cd75be9a850"
format_stamp: "Formatted at 2026-06-19 10:28:14 on dist-test-slave-hw7q"
server_key: "c0bf9ad747187d21a843f6dd901f8779"
server_key_iv: "2ecad20aa1feb30b78e808e9b60d18a4"
server_key_version: "encryptionkey@0"
I20260619 10:28:14.699889 22928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:28:14.700628 22943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:14.700776 22928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:14.700855 22928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/wal
uuid: "0ed2fede092140e0a4996cd75be9a850"
format_stamp: "Formatted at 2026-06-19 10:28:14 on dist-test-slave-hw7q"
server_key: "c0bf9ad747187d21a843f6dd901f8779"
server_key_iv: "2ecad20aa1feb30b78e808e9b60d18a4"
server_key_version: "encryptionkey@0"
I20260619 10:28:14.700939 22928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:14.709389 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } attempt: 2
I20260619 10:28:14.709836 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.709698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:44353 (local address 127.0.0.1:50226)
0619 10:28:14.709749 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.709808 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 10:28:14.710170 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.710100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:44353 (local address 127.0.0.1:50232)
0619 10:28:14.710132 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.710154 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 10:28:14.710266 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.125:44353: Network error: Client connection negotiation failed: client connection to 127.9.90.125:44353: connect: Connection refused (error 111)
I20260619 10:28:14.727495 22928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:14.729581 22928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:14.729696 22928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:14.733312 22928 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.125:44353
I20260619 10:28:14.733368 22995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.125:44353 every 8 connection(s)
I20260619 10:28:14.733654 22928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
I20260619 10:28:14.734475 22996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:14.735956 22996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:28:14.740672 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 22928
I20260619 10:28:14.740758 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-1/wal/instance
I20260619 10:28:14.740940 9577 external_mini_cluster.cc:1428] Setting key ea95b0fd6d32570b8269dcf7ba35ad53
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:14.745088 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.736676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.9.90.126:42145)
0619 10:28:14.736747 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.736749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.736865 (+ 116us) server_negotiation.cc:407] Connection header received
0619 10:28:14.737339 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.737340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.737388 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.737454 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.738066 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.738569 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.739611 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.739817 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.739880 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.742219 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.742221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.742221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.742229 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.743768 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.744146 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.744147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.744147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.744213 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.744476 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.744477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.744477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.744611 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.744681 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.744905 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.744988 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 10:28:14.745141 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.736602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42145 (local address 127.0.0.1:58816)
0619 10:28:14.736750 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.736792 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:14.737243 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:14.737516 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:14.737519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:14.737754 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:14.737968 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.737973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.738670 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.738671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.739485 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.739487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.739915 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.739916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.740031 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.740525 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:14.740533 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:14.742126 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:14.743852 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.743859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.743860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.744080 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.744272 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.744273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.744273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.744368 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.744671 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:14.744675 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:14.744789 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 10:28:14.744870 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 10:28:14.744987 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260619 10:28:14.745682 22996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.90.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.124@KRBTEST.COM" created.
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
I20260619 10:28:14.754015 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.745917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:44353 (local address 127.0.0.1:50234)
0619 10:28:14.745965 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.745977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:14.746021 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:14.746216 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:14.746217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:14.746359 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:14.746425 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.746427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.747243 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.747244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.748017 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.748019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.748474 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.748475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.748536 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.748916 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:14.748920 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:14.751409 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:14.753002 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.753003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.753004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.753172 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.753366 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.753366 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.753367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.753418 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.753691 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:14.753691 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:14.753728 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 10:28:14.753779 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:28:14.753910 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 10:28:14.754117 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.745975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.9.90.125:44353)
0619 10:28:14.746094 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.746098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.746115 (+ 17us) server_negotiation.cc:407] Connection header received
0619 10:28:14.746132 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.746134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.746144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.746201 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.746496 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.747164 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.748096 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.748378 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.748476 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.751490 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.751497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.751498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.751503 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.752908 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.753242 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.753245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.753245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.753306 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.753468 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.753468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.753468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.753631 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.753708 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.753830 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.753977 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260619 10:28:14.754937 22996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:28:14.755278 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.755154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58868)
0619 10:28:14.755201 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.755255 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 10:28:14.755435 22946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
I20260619 10:28:14.755860 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.755766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58882)
0619 10:28:14.755811 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.755839 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 10:28:14.756006 22996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:46361: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab.
Entry for principal kudu/127.9.90.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab.
I20260619 10:28:14.764835 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.124:46361
--webserver_interface=127.9.90.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab
--principal=kudu/127.9.90.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:42145,127.9.90.125:44353,127.9.90.124:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:14.801441 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } attempt: 3
I20260619 10:28:14.805236 22996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } attempt: 1
I20260619 10:28:14.805646 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.805498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58890)
0619 10:28:14.805564 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.805607 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
I20260619 10:28:14.806559 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.806454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58904)
0619 10:28:14.806497 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.806530 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:28:14.806891 22996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:46361: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
I20260619 10:28:14.809121 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.801958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.9.90.125:44353)
0619 10:28:14.802022 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.802024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.802122 (+ 98us) server_negotiation.cc:407] Connection header received
0619 10:28:14.802206 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.802207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.802218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.802261 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.802651 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.803253 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.804253 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.804497 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.804552 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.806607 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.806608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.806609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.806617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.808071 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.808393 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.808394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.808394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.808453 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.808633 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.808634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.808634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.808750 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.808810 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.808962 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.809038 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:28:14.809123 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.801998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:44353 (local address 127.0.0.1:50240)
0619 10:28:14.802058 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.802069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:14.802129 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:14.802289 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:14.802289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:14.802471 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:14.802550 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.802552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.803318 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.803319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.804163 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.804166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.804565 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.804566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.804642 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.805064 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:14.805069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:14.806533 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:14.808142 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.808143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.808144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.808318 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.808509 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.808510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.808510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.808573 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.808807 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:14.808808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:14.808859 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 10:28:14.808921 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 10:28:14.809022 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 10:28:14.809607 22921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:28:14.809963 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.809862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58914)
0619 10:28:14.809904 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.809931 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 10:28:14.810257 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.810199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58924)
0619 10:28:14.810224 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.810245 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 10:28:14.810344 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:46361: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
I20260619 10:28:14.841516 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } attempt: 1
I20260619 10:28:14.841993 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.841858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58934)
0619 10:28:14.841915 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.841958 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:28:14.842334 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.842239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58946)
0619 10:28:14.842283 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.842311 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 10:28:14.842429 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:46361: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
W20260619 10:28:14.844616 23003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:14.844787 23003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:14.844807 23003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:14.845923 23003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:14.845958 23003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:14.845970 23003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:14.845981 23003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:14.845991 23003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:14.847429 23003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:42145,127.9.90.125:44353,127.9.90.124:46361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab
--principal=kudu/127.9.90.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.124:46361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.90.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:14.847609 23003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:14.847761 23003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:14.852031 23009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:14.852005 23008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:14.852020 23011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:14.852270 23003 server_base.cc:1034] running on GCE node
I20260619 10:28:14.852480 23003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:14.852716 23003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:14.853850 23003 hybrid_clock.cc:648] HybridClock initialized: now 1781864894853836 us; error 27 us; skew 500 ppm
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:14.856123 23003 init.cc:377] Logged in from keytab as kudu/127.9.90.124@KRBTEST.COM (short username kudu)
I20260619 10:28:14.856524 23003 webserver.cc:460] Webserver started at http://127.9.90.124:38681/ using document root <none> and password file <none>
I20260619 10:28:14.856747 23003 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:14.856788 23003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:14.856905 23003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:14.857903 23003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/data/instance:
uuid: "60182641509c47ecb5b9b2222bcc697b"
format_stamp: "Formatted at 2026-06-19 10:28:14 on dist-test-slave-hw7q"
server_key: "046f11d9fe6ab3ee14deace29ac011dc"
server_key_iv: "f84704be0044e4a658d1a6e2ca142198"
server_key_version: "encryptionkey@0"
I20260619 10:28:14.858212 23003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/wal/instance:
uuid: "60182641509c47ecb5b9b2222bcc697b"
format_stamp: "Formatted at 2026-06-19 10:28:14 on dist-test-slave-hw7q"
server_key: "046f11d9fe6ab3ee14deace29ac011dc"
server_key_iv: "f84704be0044e4a658d1a6e2ca142198"
server_key_version: "encryptionkey@0"
I20260619 10:28:14.859475 23003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:14.860328 23018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:14.860505 23003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:14.860589 23003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/wal
uuid: "60182641509c47ecb5b9b2222bcc697b"
format_stamp: "Formatted at 2026-06-19 10:28:14 on dist-test-slave-hw7q"
server_key: "046f11d9fe6ab3ee14deace29ac011dc"
server_key_iv: "f84704be0044e4a658d1a6e2ca142198"
server_key_version: "encryptionkey@0"
I20260619 10:28:14.860657 23003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:14.875032 22996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } attempt: 2
I20260619 10:28:14.875466 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.875308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58952)
0619 10:28:14.875373 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.875422 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 10:28:14.876132 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.876025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58966)
0619 10:28:14.876072 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.876104 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 10:28:14.876224 22996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:46361: Network error: Client connection negotiation failed: client connection to 127.9.90.124:46361: connect: Connection refused (error 111)
I20260619 10:28:14.892369 23003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:14.894845 23003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:14.894982 23003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:14.898778 23003 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.124:46361
I20260619 10:28:14.898874 23070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.124:46361 every 8 connection(s)
I20260619 10:28:14.899219 23003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
I20260619 10:28:14.900074 23071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:14.901571 23071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:14.909266 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 23003
I20260619 10:28:14.909360 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781864754199919-9577-0/minicluster-data/master-2/wal/instance
I20260619 10:28:14.909538 9577 external_mini_cluster.cc:1428] Setting key 2e453bf3d44099c43ef486c8b0ea3bf6
I20260619 10:28:14.910171 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.902492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.9.90.126:42145)
0619 10:28:14.902562 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.902564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.902574 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:28:14.902996 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.902998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.903009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.903061 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.903725 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.904211 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.905188 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.905390 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.905445 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.907335 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.907336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.907337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.907344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.908663 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.909064 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.909066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.909066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.909136 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.909399 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.909400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.909401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.909546 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.909624 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.909965 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.910074 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 10:28:14.910625 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.902257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42145 (local address 127.0.0.1:58822)
0619 10:28:14.902394 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.902428 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:14.902902 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:14.903124 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:14.903132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:14.903406 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:14.903623 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.903627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.904288 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.904289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.905110 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.905112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.905460 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.905461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.905579 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.906002 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:14.906010 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:14.907253 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:14.908751 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.908758 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.908760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.908990 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.909194 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.909195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.909196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.909296 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.909625 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:14.909632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:14.909766 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 10:28:14.910263 (+ 497us) client_negotiation.cc:240] Negotiation successful
0619 10:28:14.910469 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260619 10:28:14.911167 23071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
I20260619 10:28:14.917208 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.911423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:44353 (local address 127.0.0.1:50250)
0619 10:28:14.911468 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.911479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:14.911522 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:14.911636 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:14.911638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:14.911781 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:14.911832 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.911833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.912395 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.912396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.912869 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.912870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.913196 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.913197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.913252 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.913657 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:14.913661 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:14.914877 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:14.916358 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.916359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.916359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.916506 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.916673 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.916673 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.916674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.916729 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.916927 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:14.916928 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:14.916969 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 10:28:14.917027 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 10:28:14.917114 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:28:14.917209 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.911482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.9.90.125:44353)
0619 10:28:14.911530 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.911532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.911543 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:28:14.911569 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.911570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.911577 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.911622 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.911888 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.912327 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.912941 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.913143 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.913195 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.914955 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.914957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.914957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.914964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.916288 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.916564 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.916566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.916566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.916619 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.916778 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.916779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.916779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.916876 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.916928 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.917055 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.917129 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:28:14.917560 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } attempt: 2
I20260619 10:28:14.917613 23071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:28:14.919171 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.909923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.9.90.124:46361)
0619 10:28:14.910098 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.910103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.910193 (+ 90us) server_negotiation.cc:407] Connection header received
0619 10:28:14.910263 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.910268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.910283 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.910340 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.911010 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.911987 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.912722 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.913038 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.913103 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.915569 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.915575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.915576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.915582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.917072 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.917480 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.917484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.917484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.917563 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.917896 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.917898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.917898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.918071 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.918207 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.919034 (+ 827us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.919102 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
I20260619 10:28:14.924888 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.917839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58990)
0619 10:28:14.917878 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.917887 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:14.917921 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:14.918201 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:14.918202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:14.918317 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:14.918344 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.918346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.919658 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.919659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.920254 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.920256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.920704 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.920705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.920744 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.921064 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:14.921069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:14.922427 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:14.923964 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.923965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.923965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.924179 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.924322 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.924322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.924323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.924372 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.924617 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:14.924618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:14.924652 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 10:28:14.924707 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 10:28:14.924794 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 10:28:14.924968 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.917975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.9.90.124:46361)
0619 10:28:14.918089 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.918090 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.918098 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:28:14.918116 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.918117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.918126 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.918194 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.918409 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.919568 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.920398 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.920634 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.920697 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.922490 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.922491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.922492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.922497 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.923885 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.924223 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.924224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.924224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.924270 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.924420 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.924421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.924422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.924565 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.924616 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.924734 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.924841 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:14.925215 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.917841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:58976)
0619 10:28:14.917889 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.917899 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:14.917937 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:14.918105 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:14.918106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:14.918253 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:14.918297 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.918300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.919005 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.919006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.919761 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.919763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.920157 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.920158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.920214 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.920636 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:14.920641 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:14.922015 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:14.924165 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.924166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.924167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.924398 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.924579 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.924579 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.924580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.924646 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.924940 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:14.924940 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:14.924983 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 10:28:14.925033 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:28:14.925116 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:28:14.925568 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.917861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.9.90.124:46361)
0619 10:28:14.917998 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.917999 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.918005 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:28:14.918022 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.918022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.918030 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.918086 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.918390 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.918930 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.919831 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.920089 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.920147 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.922116 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.922117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.922118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.922124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.924079 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.924478 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.924479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.924479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.924518 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.924725 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.924726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.924726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.924851 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.925330 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.925399 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.925473 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260619 10:28:14.926911 23071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b: Bootstrap starting.
I20260619 10:28:14.927376 22921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1: Bootstrap starting.
I20260619 10:28:14.927634 23071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:14.927959 23071 log.cc:824] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:14.928045 22921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:14.928354 22921 log.cc:824] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:14.928874 23071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b: No bootstrap required, opened a new log
I20260619 10:28:14.929056 22921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1: No bootstrap required, opened a new log
I20260619 10:28:14.931053 22921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } }
I20260619 10:28:14.931227 22921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:14.931262 22921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407c06e76b20455cb8c1b042616eaaa1, State: Initialized, Role: FOLLOWER
I20260619 10:28:14.931385 22921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } }
I20260619 10:28:14.931463 23071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } }
I20260619 10:28:14.931615 23071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:14.931656 23071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60182641509c47ecb5b9b2222bcc697b, State: Initialized, Role: FOLLOWER
I20260619 10:28:14.931700 22921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:14.931646 23080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:14.931766 23080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:28:14.931775 23071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } }
I20260619 10:28:14.932004 23081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:14.932057 23071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:14.932077 23081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:28:14.933760 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.920334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.9.90.126:42145)
0619 10:28:14.920394 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.920396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.920404 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:28:14.920424 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.920425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.920434 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.920494 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.921442 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.922005 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.923003 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.923290 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.923353 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.926268 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.926270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.926270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.926277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.927614 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.927979 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.927980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.927980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.928037 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.928285 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.928286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.928286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.928415 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.928543 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.933580 (+ 5037us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.933679 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 10:28:14.933976 23103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:28:14.934022 23103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 10:28:14.934700 23098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:28:14.934813 23098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:28:14.934748 22853 master_runner.cc:186] Error getting master registration for 127.9.90.125:44353: OK, instance_id { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" instance_seqno: 1781864894732309 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:28:14.937394 22853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
I20260619 10:28:14.937726 23003 master_runner.cc:186] Error getting master registration for 127.9.90.125:44353: OK, instance_id { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" instance_seqno: 1781864894732309 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:28:14.939931 23003 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:28:14.943312 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.934580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.9.90.125:44353)
0619 10:28:14.934617 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.934619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.934739 (+ 120us) server_negotiation.cc:407] Connection header received
0619 10:28:14.934761 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.934762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.934772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.934811 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.935382 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.936090 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.937008 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.937305 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.937364 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.940181 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.940182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.940182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.940192 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.941822 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.942190 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.942191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.942192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.942253 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.942442 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.942443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.942443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.942573 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.943101 (+ 528us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.943142 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.943219 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:28:14.944561 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:28:14.961432 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.946063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.9.90.125:44353)
0619 10:28:14.946107 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.946109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.949997 (+ 3888us) server_negotiation.cc:407] Connection header received
0619 10:28:14.950102 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.950103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.950115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.950174 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.950859 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.951668 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.952697 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.952993 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.953043 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.957560 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.957562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.957562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.957569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.960202 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.960575 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.960577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.960577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.960634 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.960819 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.960820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.960820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.960946 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.961029 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.961247 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.961330 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:28:14.961851 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.946266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.9.90.124:46361)
0619 10:28:14.946302 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.946303 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.950170 (+ 3867us) server_negotiation.cc:407] Connection header received
0619 10:28:14.950255 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.950256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.950266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.950294 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.950941 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.951804 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.952818 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.953164 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.953204 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.958055 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.958057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.958058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.958066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.960539 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.960923 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.960924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.960925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.960979 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.961228 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.961229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.961229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.961370 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.961454 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.961653 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.961736 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:28:14.965952 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.946416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.9.90.126:42145)
0619 10:28:14.946459 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.946461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.953602 (+ 7141us) server_negotiation.cc:407] Connection header received
0619 10:28:14.953717 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.953719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.953729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.953758 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.954474 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.955217 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.956232 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.956534 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.956595 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.963129 (+ 6534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.963130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.963131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.963138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.964672 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.965078 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.965079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.965080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.965140 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.965384 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.965384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.965385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.965511 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.965594 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.965745 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.965841 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:28:14.967368 22996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } attempt: 3
Jun 19 10:28:14 dist-test-slave-hw7q krb5kdc[22840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864894, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
I20260619 10:28:14.974982 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.967598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:46361 (local address 127.0.0.1:59004)
0619 10:28:14.967642 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:14.967651 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:14.967682 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:14.967907 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:14.967907 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:14.968060 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:14.968101 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.968103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.968971 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.968972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.969775 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:14.969777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.970263 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.970264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:14.970323 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.970689 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:14.970694 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:14.972171 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:14.973999 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.974000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.974000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.974209 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.974405 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:14.974406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:14.974406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:14.974465 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:14.974696 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:14.974697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:14.974739 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 10:28:14.974806 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 10:28:14.974893 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:28:14.974993 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:14.967746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.9.90.124:46361)
0619 10:28:14.967796 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:14.967798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:14.967805 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:28:14.967825 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:14.967825 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:14.967836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:14.967895 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:14.968166 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.968897 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.969867 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:14.970190 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:14.970258 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:14.972251 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:14.972253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:14.972254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:14.972260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:14.973909 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.974276 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.974276 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.974277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.974346 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:14.974518 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:14.974519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:14.974519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:14.974638 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:14.974699 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:14.974832 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 10:28:14.974903 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:28:14.976871 22996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850: Bootstrap starting.
I20260619 10:28:14.977613 22996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:14.977954 22996 log.cc:824] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:14.979212 22996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850: No bootstrap required, opened a new log
I20260619 10:28:14.981178 22996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } }
I20260619 10:28:14.981323 22996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:14.981354 22996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed2fede092140e0a4996cd75be9a850, State: Initialized, Role: FOLLOWER
I20260619 10:28:14.981474 22996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } }
I20260619 10:28:14.981813 22996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 10:28:14.983610 23125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:28:14.983709 23125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:28:14.985554 22928 master_runner.cc:263] No leader master found from master 0ed2fede092140e0a4996cd75be9a850
I20260619 10:28:14.985606 22928 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:28:14.981774 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:14.985652 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:28:15.073493 22874 tablet.cc:2369] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:28:15.116434 23080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:15.116565 23080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } }
I20260619 10:28:15.116794 23080 leader_election.cc:288] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed2fede092140e0a4996cd75be9a850 (127.9.90.125:44353), 60182641509c47ecb5b9b2222bcc697b (127.9.90.124:46361)
I20260619 10:28:15.117084 22970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "407c06e76b20455cb8c1b042616eaaa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed2fede092140e0a4996cd75be9a850" is_pre_election: true
I20260619 10:28:15.117079 23045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "407c06e76b20455cb8c1b042616eaaa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60182641509c47ecb5b9b2222bcc697b" is_pre_election: true
I20260619 10:28:15.117228 23045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 407c06e76b20455cb8c1b042616eaaa1 in term 0.
I20260619 10:28:15.117228 22970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 407c06e76b20455cb8c1b042616eaaa1 in term 0.
I20260619 10:28:15.117408 22871 leader_election.cc:302] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 407c06e76b20455cb8c1b042616eaaa1, 60182641509c47ecb5b9b2222bcc697b; no voters:
I20260619 10:28:15.117574 23080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:28:15.117678 23080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:28:15.117703 23080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:15.118631 23080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } }
I20260619 10:28:15.118779 23080 leader_election.cc:288] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed2fede092140e0a4996cd75be9a850 (127.9.90.125:44353), 60182641509c47ecb5b9b2222bcc697b (127.9.90.124:46361)
I20260619 10:28:15.118959 23045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "407c06e76b20455cb8c1b042616eaaa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60182641509c47ecb5b9b2222bcc697b"
I20260619 10:28:15.119035 23045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:15.119035 22970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "407c06e76b20455cb8c1b042616eaaa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed2fede092140e0a4996cd75be9a850"
I20260619 10:28:15.119107 22970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:15.119808 23045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 407c06e76b20455cb8c1b042616eaaa1 in term 1.
I20260619 10:28:15.119863 22970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 407c06e76b20455cb8c1b042616eaaa1 in term 1.
I20260619 10:28:15.120002 22871 leader_election.cc:302] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 407c06e76b20455cb8c1b042616eaaa1, 60182641509c47ecb5b9b2222bcc697b; no voters:
I20260619 10:28:15.120087 23080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:28:15.120191 23080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [term 1 LEADER]: Becoming Leader. State: Replica: 407c06e76b20455cb8c1b042616eaaa1, State: Running, Role: LEADER
I20260619 10:28:15.120278 23080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } }
I20260619 10:28:15.120653 23127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 407c06e76b20455cb8c1b042616eaaa1. Latest consensus state: current_term: 1 leader_uuid: "407c06e76b20455cb8c1b042616eaaa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:15.120772 23127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:15.121003 23129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:28:15.121202 23129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:28:15.122462 22970 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [term 1 FOLLOWER]: Refusing update from remote peer 407c06e76b20455cb8c1b042616eaaa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:15.122504 23045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [term 1 FOLLOWER]: Refusing update from remote peer 407c06e76b20455cb8c1b042616eaaa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:15.122735 23127 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:15.122849 23080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:15.123968 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 407c06e76b20455cb8c1b042616eaaa1. Latest consensus state: current_term: 1 leader_uuid: "407c06e76b20455cb8c1b042616eaaa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:15.124051 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:28:15.125070 23081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 407c06e76b20455cb8c1b042616eaaa1. Latest consensus state: current_term: 1 leader_uuid: "407c06e76b20455cb8c1b042616eaaa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:15.125156 23081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:28:15.125485 23080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "407c06e76b20455cb8c1b042616eaaa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:15.125559 23080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:15.125609 23130 mvcc.cc:205] Tried to move back new op lower bound from 7298518610419650560 to 7298518610413211648. Current Snapshot: MvccSnapshot[applied={T|T < 7298518610419650560}]
I20260619 10:28:15.125650 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "407c06e76b20455cb8c1b042616eaaa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:15.125723 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:28:15.125986 23129 catalog_manager.cc:1347] Generated new cluster ID: 432f3ef59c6c404a99e409f0694c8282
I20260619 10:28:15.126032 23129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:28:15.130086 23131 mvcc.cc:205] Tried to move back new op lower bound from 7298518610419650560 to 7298518610413211648. Current Snapshot: MvccSnapshot[applied={T|T < 7298518610419650560 or (T in {7298518610419650560})}]
I20260619 10:28:15.130282 23127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "407c06e76b20455cb8c1b042616eaaa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:15.130347 23127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1 [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:15.130714 23081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "407c06e76b20455cb8c1b042616eaaa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "407c06e76b20455cb8c1b042616eaaa1" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42145 } } peers { permanent_uuid: "0ed2fede092140e0a4996cd75be9a850" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 44353 } } peers { permanent_uuid: "60182641509c47ecb5b9b2222bcc697b" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 46361 } } }
I20260619 10:28:15.130784 23081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:28:15.133816 23129 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:28:15.134315 23129 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:28:15.138425 23129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 407c06e76b20455cb8c1b042616eaaa1: Generated new TSK 0
I20260619 10:28:15.138587 23129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:28:15.934432 23103 catalog_manager.cc:1259] Loaded cluster ID: 432f3ef59c6c404a99e409f0694c8282
I20260619 10:28:15.934516 23103 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b: loading cluster ID for follower catalog manager: success
I20260619 10:28:15.935305 23103 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b: acquiring CA information for follower catalog manager: success
I20260619 10:28:15.935575 23103 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60182641509c47ecb5b9b2222bcc697b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:28:15.984220 23125 catalog_manager.cc:1259] Loaded cluster ID: 432f3ef59c6c404a99e409f0694c8282
I20260619 10:28:15.984290 23125 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850: loading cluster ID for follower catalog manager: success
I20260619 10:28:15.985052 23125 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850: acquiring CA information for follower catalog manager: success
I20260619 10:28:15.985316 23125 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ed2fede092140e0a4996cd75be9a850: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:28:16.191459 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 22853
I20260619 10:28:16.195686 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 22928
I20260619 10:28:16.199290 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 23003
2026-06-19T10:28:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2786 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:28:16 dist-test-slave-hw7q krb5kdc[23146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:28:16 dist-test-slave-hw7q krb5kdc[23146](info): set up 2 sockets
Jun 19 10:28:16 dist-test-slave-hw7q krb5kdc[23146](info): commencing operation
krb5kdc: starting...
W20260619 10:28:17.149706 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:28:17 dist-test-slave-hw7q krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:28:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:28:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:28:17.259974 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:44759
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:36527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:44759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:17.336123 23159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:17.336318 23159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:17.336349 23159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:17.337462 23159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:17.337502 23159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:17.337520 23159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:17.337536 23159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:17.337553 23159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:17.338979 23159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:36527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:44759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:44759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:17.339203 23159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:17.339395 23159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:17.343915 23167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:17.343945 23159 server_base.cc:1034] running on GCE node
W20260619 10:28:17.343892 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:17.343909 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:17.344336 23159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:17.344626 23159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:17.345775 23159 hybrid_clock.cc:648] HybridClock initialized: now 1781864897345761 us; error 30 us; skew 500 ppm
Jun 19 10:28:17 dist-test-slave-hw7q krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864897, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:17.348021 23159 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:28:17.348381 23159 webserver.cc:460] Webserver started at http://127.9.90.126:41761/ using document root <none> and password file <none>
I20260619 10:28:17.348580 23159 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:17.348627 23159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:17.348714 23159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:17.349740 23159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "abde95a4bfe0407cb326d455159827e0"
format_stamp: "Formatted at 2026-06-19 10:28:17 on dist-test-slave-hw7q"
server_key: "60b02b8ac93241107e1df627cb4ad3e7"
server_key_iv: "6583c22eceb515ebe6a31902ec6b7367"
server_key_version: "encryptionkey@0"
I20260619 10:28:17.350193 23159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "abde95a4bfe0407cb326d455159827e0"
format_stamp: "Formatted at 2026-06-19 10:28:17 on dist-test-slave-hw7q"
server_key: "60b02b8ac93241107e1df627cb4ad3e7"
server_key_iv: "6583c22eceb515ebe6a31902ec6b7367"
server_key_version: "encryptionkey@0"
I20260619 10:28:17.351490 23159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:17.352164 23174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:17.352281 23159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:17.352348 23159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "abde95a4bfe0407cb326d455159827e0"
format_stamp: "Formatted at 2026-06-19 10:28:17 on dist-test-slave-hw7q"
server_key: "60b02b8ac93241107e1df627cb4ad3e7"
server_key_iv: "6583c22eceb515ebe6a31902ec6b7367"
server_key_version: "encryptionkey@0"
I20260619 10:28:17.352401 23159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:17.360635 23159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:17.362404 23159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:17.362546 23159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:17.366082 23159 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:44759
I20260619 10:28:17.366112 23226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:44759 every 8 connection(s)
I20260619 10:28:17.366420 23159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:28:17.367259 23227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:17.369463 23227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0: Bootstrap starting.
I20260619 10:28:17.370019 23227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:17.370249 23227 log.cc:824] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:17.370940 23227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0: No bootstrap required, opened a new log
I20260619 10:28:17.372493 23227 raft_consensus.cc:348] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abde95a4bfe0407cb326d455159827e0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44759 } }
I20260619 10:28:17.372627 23227 raft_consensus.cc:374] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:17.372646 23227 raft_consensus.cc:729] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abde95a4bfe0407cb326d455159827e0, State: Initialized, Role: FOLLOWER
I20260619 10:28:17.372749 23227 consensus_queue.cc:260] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abde95a4bfe0407cb326d455159827e0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44759 } }
I20260619 10:28:17.372802 23227 raft_consensus.cc:388] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:28:17.372830 23227 raft_consensus.cc:482] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:28:17.372869 23227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:17.373358 23227 raft_consensus.cc:504] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abde95a4bfe0407cb326d455159827e0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44759 } }
I20260619 10:28:17.373459 23227 leader_election.cc:302] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abde95a4bfe0407cb326d455159827e0; no voters:
I20260619 10:28:17.373605 23227 leader_election.cc:288] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:28:17.373661 23230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:28:17.373891 23227 sys_catalog.cc:564] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:17.373878 23230 raft_consensus.cc:686] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [term 1 LEADER]: Becoming Leader. State: Replica: abde95a4bfe0407cb326d455159827e0, State: Running, Role: LEADER
I20260619 10:28:17.374023 23230 consensus_queue.cc:237] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abde95a4bfe0407cb326d455159827e0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44759 } }
I20260619 10:28:17.374063 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 23159
I20260619 10:28:17.374166 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:28:17.374367 9577 external_mini_cluster.cc:1428] Setting key 4a9a01a0e3186b3a5437dc0de160f9cd
I20260619 10:28:17.374488 23232 sys_catalog.cc:455] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abde95a4bfe0407cb326d455159827e0. Latest consensus state: current_term: 1 leader_uuid: "abde95a4bfe0407cb326d455159827e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abde95a4bfe0407cb326d455159827e0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44759 } } }
I20260619 10:28:17.374567 23232 sys_catalog.cc:458] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:17.374609 23231 sys_catalog.cc:455] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abde95a4bfe0407cb326d455159827e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abde95a4bfe0407cb326d455159827e0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 44759 } } }
I20260619 10:28:17.374655 23231 sys_catalog.cc:458] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0 [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:17.374902 23237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:28:17.375401 23237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:28:17.377015 23237 catalog_manager.cc:1347] Generated new cluster ID: e020cd3dd62f4162bce908d98526ef62
I20260619 10:28:17.377065 23237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:28:17 dist-test-slave-hw7q krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:17.383898 23237 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:28:17.384371 23237 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:28:17.385859 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:17.374906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.9.90.126:44759)
0619 10:28:17.375081 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:17.375092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:17.375269 (+ 177us) server_negotiation.cc:407] Connection header received
0619 10:28:17.375515 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:17.375521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:17.375712 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:17.375834 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:17.376510 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:17.377858 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:17.378918 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:17.379522 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:17.379614 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:17.382336 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:17.382340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:17.382340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:17.382352 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:17.384271 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:17.384656 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:17.384659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:17.384660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:17.384772 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:17.384950 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:17.384951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:17.384951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:17.385224 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:17.385376 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:17.385481 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 10:28:17.385628 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260619 10:28:17.387001 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:28:17.391924 23237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abde95a4bfe0407cb326d455159827e0: Generated new TSK 0
I20260619 10:28:17.392086 23237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:28:17.440973 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 23159
2026-06-19T10:28:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1240 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:28:17 dist-test-slave-hw7q krb5kdc[23254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:28:17 dist-test-slave-hw7q krb5kdc[23254](info): set up 2 sockets
Jun 19 10:28:17 dist-test-slave-hw7q krb5kdc[23254](info): commencing operation
krb5kdc: starting...
W20260619 10:28:18.377652 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:28:18 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:28:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:28:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:28:18.484773 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33143
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:18.561116 23267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:18.561308 23267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:18.561329 23267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:18.562438 23267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:18.562474 23267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:18.562486 23267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:18.562497 23267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:18.562507 23267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:18.563854 23267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:33143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:18.564023 23267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:18.564186 23267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:18.568552 23273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:18.568593 23275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:18.568635 23267 server_base.cc:1034] running on GCE node
W20260619 10:28:18.568606 23272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:18.568933 23267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:18.569162 23267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:18.570322 23267 hybrid_clock.cc:648] HybridClock initialized: now 1781864898570311 us; error 29 us; skew 500 ppm
Jun 19 10:28:18 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864898, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:18.572607 23267 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:28:18.572983 23267 webserver.cc:460] Webserver started at http://127.9.90.126:34787/ using document root <none> and password file <none>
I20260619 10:28:18.573158 23267 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:18.573199 23267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:18.573303 23267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:18.574304 23267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "739bc8f60fd44e729c538f6d920227bc"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "8dc88460aef5b34d9d259f15cfbbe5e4"
server_key_iv: "61c2618f61c3fed401c8efef8127849a"
server_key_version: "encryptionkey@0"
I20260619 10:28:18.574620 23267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "739bc8f60fd44e729c538f6d920227bc"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "8dc88460aef5b34d9d259f15cfbbe5e4"
server_key_iv: "61c2618f61c3fed401c8efef8127849a"
server_key_version: "encryptionkey@0"
I20260619 10:28:18.575798 23267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:28:18.576507 23282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:18.576653 23267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:18.576732 23267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "739bc8f60fd44e729c538f6d920227bc"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "8dc88460aef5b34d9d259f15cfbbe5e4"
server_key_iv: "61c2618f61c3fed401c8efef8127849a"
server_key_version: "encryptionkey@0"
I20260619 10:28:18.576800 23267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:18.596717 23267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:18.599013 23267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:18.599139 23267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:18.603232 23267 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:33143
I20260619 10:28:18.603369 23334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:33143 every 8 connection(s)
I20260619 10:28:18.603636 23267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:28:18.604611 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:18.606866 23335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Bootstrap starting.
I20260619 10:28:18.607416 23335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:18.607663 23335 log.cc:824] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:18.608397 23335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: No bootstrap required, opened a new log
I20260619 10:28:18.609048 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 23267
I20260619 10:28:18.609139 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:28:18.609315 9577 external_mini_cluster.cc:1428] Setting key a7e2ae4a84df9967b70fb53fe591cfce
I20260619 10:28:18.610399 23335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:18.610564 23335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:18.610599 23335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739bc8f60fd44e729c538f6d920227bc, State: Initialized, Role: FOLLOWER
I20260619 10:28:18.610762 23335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:18.610827 23335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:28:18.610847 23335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:28:18.610891 23335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:18.611889 23335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:18.612000 23335 leader_election.cc:302] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 739bc8f60fd44e729c538f6d920227bc; no voters:
I20260619 10:28:18.612146 23335 leader_election.cc:288] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:28:18.612191 23340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:28:18.612367 23335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:18.612385 23340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 1 LEADER]: Becoming Leader. State: Replica: 739bc8f60fd44e729c538f6d920227bc, State: Running, Role: LEADER
I20260619 10:28:18.612556 23340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:18.612933 23341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "739bc8f60fd44e729c538f6d920227bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } } }
I20260619 10:28:18.613029 23341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:18.613168 23342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 739bc8f60fd44e729c538f6d920227bc. Latest consensus state: current_term: 1 leader_uuid: "739bc8f60fd44e729c538f6d920227bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } } }
I20260619 10:28:18.613219 23342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:18.613502 23349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:28:18.613993 23349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:28:18.616118 23349 catalog_manager.cc:1347] Generated new cluster ID: 734d1addbc2a445fa5aa219bf4407205
I20260619 10:28:18.616175 23349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:28:18 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:18.619549 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:18.609861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.9.90.126:33143)
0619 10:28:18.610071 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:18.610080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:18.610114 (+ 34us) server_negotiation.cc:407] Connection header received
0619 10:28:18.610368 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:18.610377 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:18.610439 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:18.610597 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:18.611247 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.612562 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.613655 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.614293 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.614420 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:18.616759 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:18.616762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:18.616762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:18.616772 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:18.618213 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:18.618510 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:18.618512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:18.618513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:18.618616 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:18.618782 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:18.618783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:18.618783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:18.618999 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:18.619134 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:18.619238 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 10:28:18.619363 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":110,"threads_started":1}
I20260619 10:28:18.646113 23349 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:28:18.646832 23349 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:28:18.655669 23349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Generated new TSK 0
I20260619 10:28:18.655851 23349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:28:18.679920 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33143
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:18.757769 23361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:18.757987 23361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:18.758016 23361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:18.759166 23361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:18.759208 23361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:18.759243 23361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:28:18.760643 23361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:18.760857 23361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:18.761052 23361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:18.765595 23367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:18.765658 23366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:18.765714 23361 server_base.cc:1034] running on GCE node
W20260619 10:28:18.765614 23369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:18.766002 23361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:18.766227 23361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:18.767400 23361 hybrid_clock.cc:648] HybridClock initialized: now 1781864898767363 us; error 31 us; skew 500 ppm
Jun 19 10:28:18 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864898, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:18.769768 23361 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:28:18.770283 23361 webserver.cc:460] Webserver started at http://127.9.90.65:39561/ using document root <none> and password file <none>
I20260619 10:28:18.770511 23361 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:18.770563 23361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:18.770689 23361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:18.771683 23361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "8df915e826664b1cb576eba53660a498"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "269d4f313016fc6037f2267e083d07ca"
server_key_iv: "a3c48282387f5eee594fd51b6de88817"
server_key_version: "encryptionkey@0"
I20260619 10:28:18.772015 23361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "8df915e826664b1cb576eba53660a498"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "269d4f313016fc6037f2267e083d07ca"
server_key_iv: "a3c48282387f5eee594fd51b6de88817"
server_key_version: "encryptionkey@0"
I20260619 10:28:18.773263 23361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:28:18.773998 23376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:18.774194 23361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:18.774283 23361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "8df915e826664b1cb576eba53660a498"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "269d4f313016fc6037f2267e083d07ca"
server_key_iv: "a3c48282387f5eee594fd51b6de88817"
server_key_version: "encryptionkey@0"
I20260619 10:28:18.774348 23361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:18.796605 23361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:18.798535 23361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:18.798646 23361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:18.798835 23361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:18.799103 23361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:18.799131 23361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:18.799152 23361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:18.799163 23361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:18.804440 23361 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:40015
I20260619 10:28:18.804515 23489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:40015 every 8 connection(s)
I20260619 10:28:18.804786 23361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:28:18 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864898, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:18.813359 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:18.805457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:54185 (local address 127.9.90.126:33143)
0619 10:28:18.805502 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:18.805504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:18.805684 (+ 180us) server_negotiation.cc:407] Connection header received
0619 10:28:18.806203 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:18.806204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:18.806219 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:18.806271 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:18.807201 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.807701 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.808676 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.808891 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.808946 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:18.811015 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:18.811016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:18.811017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:18.811026 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:18.812117 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:18.812479 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:18.812481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:18.812481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:18.812532 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:18.812735 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:18.812736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:18.812736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:18.812840 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:18.812905 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:18.813164 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 10:28:18.813253 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:28:18.813536 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:18.805396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:54185)
0619 10:28:18.805569 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:18.805607 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:18.806105 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:18.806352 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:18.806356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:18.806921 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:18.807105 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:18.807109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.807780 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.807782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:18.808591 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:18.808594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.808959 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.808960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:18.809113 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:18.809571 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:18.809579 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:18.810925 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:18.812187 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:18.812194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:18.812196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:18.812413 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:18.812584 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:18.812585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:18.812586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:18.812681 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:18.812913 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:18.812920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:18.813048 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 10:28:18.813133 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 10:28:18.813340 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260619 10:28:18.813987 23490 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33143
I20260619 10:28:18.814069 23490 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:18.814208 23490 heartbeater.cc:511] Master 127.9.90.126:33143 requested a full tablet report, sending...
I20260619 10:28:18.814651 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 23361
I20260619 10:28:18.814733 23299 ts_manager.cc:194] Registered new tserver with Master: 8df915e826664b1cb576eba53660a498 (127.9.90.65:40015)
I20260619 10:28:18.814757 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:28:18.814957 9577 external_mini_cluster.cc:1428] Setting key 0cb7651b1a3cd64a1dd80c5422172de0
I20260619 10:28:18.815515 23299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:54185
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:28:18.836560 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33143
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:18.913025 23495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:18.913223 23495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:18.913252 23495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:18.914402 23495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:18.914441 23495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:18.914475 23495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:28:18.915840 23495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:18.916052 23495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:18.916242 23495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:18.920677 23500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:18.920686 23503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:18.920686 23501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:18.920954 23495 server_base.cc:1034] running on GCE node
I20260619 10:28:18.921130 23495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:18.921339 23495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:18.922472 23495 hybrid_clock.cc:648] HybridClock initialized: now 1781864898922456 us; error 32 us; skew 500 ppm
Jun 19 10:28:18 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864898, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:18.924817 23495 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:28:18.925220 23495 webserver.cc:460] Webserver started at http://127.9.90.66:45029/ using document root <none> and password file <none>
I20260619 10:28:18.925428 23495 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:18.925475 23495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:18.925603 23495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:18.926617 23495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "d247c2f578814bacabf2c523d5c2896e"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "b881393b9821dd933cb15341020db0f3"
server_key_iv: "5e6171ca92246fe25deeefcb64781e20"
server_key_version: "encryptionkey@0"
I20260619 10:28:18.926941 23495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "d247c2f578814bacabf2c523d5c2896e"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "b881393b9821dd933cb15341020db0f3"
server_key_iv: "5e6171ca92246fe25deeefcb64781e20"
server_key_version: "encryptionkey@0"
I20260619 10:28:18.928107 23495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:28:18.928799 23510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:18.928953 23495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:18.929025 23495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "d247c2f578814bacabf2c523d5c2896e"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "b881393b9821dd933cb15341020db0f3"
server_key_iv: "5e6171ca92246fe25deeefcb64781e20"
server_key_version: "encryptionkey@0"
I20260619 10:28:18.929088 23495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:18.941941 23495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:18.943858 23495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:18.943990 23495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:18.944188 23495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:18.944473 23495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:18.944504 23495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:18.944531 23495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:18.944555 23495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:18.949685 23495 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:43907
I20260619 10:28:18.949744 23623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:43907 every 8 connection(s)
I20260619 10:28:18.950093 23495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:28:18.950541 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 23495
I20260619 10:28:18.950632 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:28:18.950824 9577 external_mini_cluster.cc:1428] Setting key 92ab1311b20bf7b9169b796b28279ad9
Jun 19 10:28:18 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864898, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
I20260619 10:28:18.959941 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:18.951030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:55025 (local address 127.9.90.126:33143)
0619 10:28:18.951071 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:18.951073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:18.951162 (+ 89us) server_negotiation.cc:407] Connection header received
0619 10:28:18.951774 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:18.951776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:18.951788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:18.951840 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:18.952801 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.953284 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.954709 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.954940 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.955010 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:18.956983 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:18.956984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:18.956985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:18.956993 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:18.958583 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:18.958995 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:18.958996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:18.958996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:18.959060 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:18.959307 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:18.959308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:18.959309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:18.959414 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:18.959545 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:18.959804 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 10:28:18.959875 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:28:18.960103 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:18.950856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:55025)
0619 10:28:18.951037 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:18.951085 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:18.951673 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:18.951900 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:18.951910 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:18.952452 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:18.952710 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:18.952717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.953388 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.953389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:18.954601 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:18.954604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:18.955026 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:18.955027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:18.955167 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:18.955621 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:18.955629 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:18.956889 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:18.958675 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:18.958682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:18.958684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:18.958913 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:18.959141 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:18.959142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:18.959142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:18.959248 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:18.959546 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:18.959550 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:18.959684 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 10:28:18.959770 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 10:28:18.959907 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260619 10:28:18.960633 23624 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33143
I20260619 10:28:18.960721 23624 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:18.960865 23624 heartbeater.cc:511] Master 127.9.90.126:33143 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
I20260619 10:28:18.961252 23299 ts_manager.cc:194] Registered new tserver with Master: d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907)
I20260619 10:28:18.961712 23299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:55025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:28:18.975044 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33143
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:19.054265 23629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:19.054467 23629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:19.054497 23629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:19.055611 23629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:19.055652 23629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:19.055686 23629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:28:19.057077 23629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:33143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:19.057294 23629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:19.057485 23629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:19.062110 23635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.062064 23634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:19.062116 23629 server_base.cc:1034] running on GCE node
W20260619 10:28:19.062080 23637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:19.062496 23629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:19.062726 23629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:19.063861 23629 hybrid_clock.cc:648] HybridClock initialized: now 1781864899063847 us; error 28 us; skew 500 ppm
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:19.066159 23629 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:28:19.066524 23629 webserver.cc:460] Webserver started at http://127.9.90.67:40369/ using document root <none> and password file <none>
I20260619 10:28:19.066692 23629 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:19.066730 23629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:19.066834 23629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:19.067770 23629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "f288406707a747eb93376f4e569ea38e"
format_stamp: "Formatted at 2026-06-19 10:28:19 on dist-test-slave-hw7q"
server_key: "2e95bfe5d683972a52af332414c6275e"
server_key_iv: "cc170edac3c1a89aaafcfde6c3eeecce"
server_key_version: "encryptionkey@0"
I20260619 10:28:19.068056 23629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "f288406707a747eb93376f4e569ea38e"
format_stamp: "Formatted at 2026-06-19 10:28:19 on dist-test-slave-hw7q"
server_key: "2e95bfe5d683972a52af332414c6275e"
server_key_iv: "cc170edac3c1a89aaafcfde6c3eeecce"
server_key_version: "encryptionkey@0"
I20260619 10:28:19.069190 23629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:28:19.069886 23644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.070051 23629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:19.070137 23629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "f288406707a747eb93376f4e569ea38e"
format_stamp: "Formatted at 2026-06-19 10:28:19 on dist-test-slave-hw7q"
server_key: "2e95bfe5d683972a52af332414c6275e"
server_key_iv: "cc170edac3c1a89aaafcfde6c3eeecce"
server_key_version: "encryptionkey@0"
I20260619 10:28:19.070205 23629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:19.083407 23629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:19.085294 23629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:19.085461 23629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:19.085657 23629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:19.085983 23629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:19.086021 23629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.086050 23629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:19.086072 23629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.091355 23629 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:34879
I20260619 10:28:19.091418 23757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:34879 every 8 connection(s)
I20260619 10:28:19.091687 23629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:19.099619 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 23629
I20260619 10:28:19.099730 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:28:19.099924 9577 external_mini_cluster.cc:1428] Setting key 04bf95cffca9bd007885190e3eec0d74
I20260619 10:28:19.100455 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.092316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:52885 (local address 127.9.90.126:33143)
0619 10:28:19.092391 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.092393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.092546 (+ 153us) server_negotiation.cc:407] Connection header received
0619 10:28:19.092982 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.092983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.092991 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.093038 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:19.093889 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.094376 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.095343 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.095542 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.095602 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.097389 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:19.097391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:19.097391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:19.097399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:19.098817 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.099196 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.099197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.099197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.099259 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.099541 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.099542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.099543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.099693 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:19.099770 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.100264 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.100360 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 10:28:19.100621 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.092290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:52885)
0619 10:28:19.092436 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.092469 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.092910 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.093097 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.093100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.093612 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:19.093806 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.093813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.094444 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.094445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.095259 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.095261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.095607 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.095607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.095695 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.096075 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:19.096082 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:19.097297 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:19.098893 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.098898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.098900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.099098 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.099345 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.099346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.099346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.099452 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.099800 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:19.099806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:19.099981 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.100224 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.100400 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260619 10:28:19.101011 23758 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33143
I20260619 10:28:19.101109 23758 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:19.101274 23758 heartbeater.cc:511] Master 127.9.90.126:33143 requested a full tablet report, sending...
I20260619 10:28:19.101613 23299 ts_manager.cc:194] Registered new tserver with Master: f288406707a747eb93376f4e569ea38e (127.9.90.67:34879)
I20260619 10:28:19.102017 23299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:52885
I20260619 10:28:19.111850 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:28:19.118742 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.113130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.9.90.126:33143)
0619 10:28:19.113175 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.113177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.113301 (+ 124us) server_negotiation.cc:407] Connection header received
0619 10:28:19.113370 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.113370 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.113381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.113425 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:19.114050 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.114526 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.115236 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.115423 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.115475 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.116579 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:19.116580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:19.116581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:19.116588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:19.117705 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.118023 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.118024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.118024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.118078 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.118235 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.118236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.118237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.118378 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:19.118463 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.118583 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.118655 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:28:19.120151 23299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:28:19.121655 23299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:28:19.127638 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.123604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:40015 (local address 127.0.0.1:42656)
0619 10:28:19.123765 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.123800 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.123858 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.124147 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.124150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.124152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:19.124213 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.124214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.125500 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.125501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.126708 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.126709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.127359 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.127360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.127435 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.127449 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.127513 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.127564 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":6,"thread_start_us":63,"threads_started":1}
I20260619 10:28:19.127724 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.123859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.9.90.66:43907)
0619 10:28:19.124003 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.124008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.124252 (+ 244us) server_negotiation.cc:407] Connection header received
0619 10:28:19.124322 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.124325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.124376 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.124448 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:19.124678 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.125658 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.126703 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.127396 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.127480 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.127535 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.127605 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.127643 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260619 10:28:19.127849 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.124029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.9.90.67:34879)
0619 10:28:19.124179 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.124182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.124193 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:28:19.124232 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.124234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.124249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.124427 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:19.124607 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.125273 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.126773 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.127423 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.127480 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.127535 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.127709 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.127761 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260619 10:28:19.128086 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.123827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.9.90.65:40015)
0619 10:28:19.124003 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.124008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.124018 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:28:19.124054 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.124056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.124070 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.124136 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:19.124287 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.125383 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.126804 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.127277 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.127338 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.127397 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.127833 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.127878 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260619 10:28:19.127985 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.123607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:34879 (local address 127.0.0.1:42692)
0619 10:28:19.123936 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.123944 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.123978 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.124507 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.124508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.124511 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:19.124553 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.124555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.125363 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.125364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.126706 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.126709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.127569 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.127571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.127632 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.127636 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.127687 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.127910 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":24,"thread_start_us":57,"threads_started":1}
I20260619 10:28:19.128007 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.123607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:43907 (local address 127.0.0.1:50180)
0619 10:28:19.124180 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.124190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.124250 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.124521 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.124522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.124524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:19.124599 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.124601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.125737 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.125737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.126634 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.126636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.127481 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.127481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.127544 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.127547 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.127592 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.127622 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":23,"thread_start_us":59,"threads_started":1}
I20260619 10:28:19.128845 23558 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1c575e592947e79942d1b1727eb8ed (DEFAULT_TABLE table=test-table [id=cef599b7375c43058b632f5271a445b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:19.128850 23424 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1c575e592947e79942d1b1727eb8ed (DEFAULT_TABLE table=test-table [id=cef599b7375c43058b632f5271a445b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:19.128964 23692 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1c575e592947e79942d1b1727eb8ed (DEFAULT_TABLE table=test-table [id=cef599b7375c43058b632f5271a445b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:19.129109 23558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1c575e592947e79942d1b1727eb8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:19.129165 23424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1c575e592947e79942d1b1727eb8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:19.129205 23692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1c575e592947e79942d1b1727eb8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:19.131744 23772 tablet_bootstrap.cc:492] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: Bootstrap starting.
I20260619 10:28:19.132222 23773 tablet_bootstrap.cc:492] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Bootstrap starting.
I20260619 10:28:19.132382 23772 tablet_bootstrap.cc:654] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:19.132833 23773 tablet_bootstrap.cc:654] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:19.132987 23772 log.cc:824] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:19.133325 23773 log.cc:824] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:19.133656 23774 tablet_bootstrap.cc:492] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: Bootstrap starting.
I20260619 10:28:19.133689 23772 tablet_bootstrap.cc:492] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: No bootstrap required, opened a new log
I20260619 10:28:19.133740 23772 ts_tablet_manager.cc:1397] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:28:19.134038 23773 tablet_bootstrap.cc:492] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: No bootstrap required, opened a new log
I20260619 10:28:19.134096 23773 ts_tablet_manager.cc:1397] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:28:19.134298 23774 tablet_bootstrap.cc:654] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:19.134799 23774 log.cc:824] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:19.135469 23772 raft_consensus.cc:348] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.135480 23773 raft_consensus.cc:348] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.135602 23772 raft_consensus.cc:374] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:19.135603 23773 raft_consensus.cc:374] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:19.135625 23772 raft_consensus.cc:729] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d247c2f578814bacabf2c523d5c2896e, State: Initialized, Role: FOLLOWER
I20260619 10:28:19.135633 23773 raft_consensus.cc:729] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df915e826664b1cb576eba53660a498, State: Initialized, Role: FOLLOWER
I20260619 10:28:19.135721 23772 consensus_queue.cc:260] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.135763 23773 consensus_queue.cc:260] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.135973 23772 ts_tablet_manager.cc:1428] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:28:19.136021 23773 ts_tablet_manager.cc:1428] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:28:19.136044 23490 heartbeater.cc:503] Master 127.9.90.126:33143 was elected leader, sending a full tablet report...
I20260619 10:28:19.136124 23624 heartbeater.cc:503] Master 127.9.90.126:33143 was elected leader, sending a full tablet report...
I20260619 10:28:19.136543 23774 tablet_bootstrap.cc:492] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: No bootstrap required, opened a new log
I20260619 10:28:19.136608 23774 ts_tablet_manager.cc:1397] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 10:28:19.138079 23774 raft_consensus.cc:348] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.138247 23774 raft_consensus.cc:374] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:19.138293 23774 raft_consensus.cc:729] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f288406707a747eb93376f4e569ea38e, State: Initialized, Role: FOLLOWER
I20260619 10:28:19.138389 23774 consensus_queue.cc:260] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.138592 23774 ts_tablet_manager.cc:1428] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:28:19.138639 23758 heartbeater.cc:503] Master 127.9.90.126:33143 was elected leader, sending a full tablet report...
W20260619 10:28:19.201122 23625 tablet.cc:2369] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:28:19.223112 23778 raft_consensus.cc:482] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:19.223240 23778 raft_consensus.cc:504] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.223574 23778 leader_election.cc:288] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df915e826664b1cb576eba53660a498 (127.9.90.65:40015), f288406707a747eb93376f4e569ea38e (127.9.90.67:34879)
I20260619 10:28:19.226250 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.223820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:47469 (local address 127.9.90.65:40015)
0619 10:28:19.223867 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.223869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.223879 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:28:19.223908 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.223909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.223920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.223982 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:19.224143 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.224614 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.225526 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.225944 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.226006 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.226017 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.226152 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.226186 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:28:19.226282 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.223730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:40015 (local address 127.9.90.66:47469)
0619 10:28:19.223776 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.223787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.223825 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.224004 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.224005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.224007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:19.224076 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.224078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.224689 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.224690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.225422 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.225424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.226014 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.226014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.226093 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.226096 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.226141 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.226180 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 10:28:19.226293 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.223812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:44489 (local address 127.9.90.67:34879)
0619 10:28:19.223872 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.223874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.224017 (+ 143us) server_negotiation.cc:407] Connection header received
0619 10:28:19.224076 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.224077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.224089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.224133 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:19.224267 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.224713 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.225588 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.225999 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.226060 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.226069 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.226193 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.226226 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 10:28:19.226377 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.223730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:34879 (local address 127.9.90.66:44489)
0619 10:28:19.223949 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.223956 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.224023 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.224138 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.224139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.224141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:19.224208 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.224210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.224779 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.224779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.225498 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.225500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.226067 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.226068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.226135 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.226138 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.226182 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.226219 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":52,"thread_start_us":111,"threads_started":1}
I20260619 10:28:19.226671 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1c575e592947e79942d1b1727eb8ed" candidate_uuid: "d247c2f578814bacabf2c523d5c2896e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f288406707a747eb93376f4e569ea38e" is_pre_election: true
I20260619 10:28:19.226683 23444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1c575e592947e79942d1b1727eb8ed" candidate_uuid: "d247c2f578814bacabf2c523d5c2896e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df915e826664b1cb576eba53660a498" is_pre_election: true
I20260619 10:28:19.226828 23444 raft_consensus.cc:2454] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d247c2f578814bacabf2c523d5c2896e in term 0.
I20260619 10:28:19.226828 23712 raft_consensus.cc:2454] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d247c2f578814bacabf2c523d5c2896e in term 0.
I20260619 10:28:19.227020 23513 leader_election.cc:302] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d247c2f578814bacabf2c523d5c2896e, f288406707a747eb93376f4e569ea38e; no voters:
I20260619 10:28:19.227141 23778 raft_consensus.cc:2781] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:28:19.227192 23778 raft_consensus.cc:482] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:28:19.227221 23778 raft_consensus.cc:3037] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:19.227871 23778 raft_consensus.cc:504] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.227977 23778 leader_election.cc:288] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [CANDIDATE]: Term 1 election: Requested vote from peers 8df915e826664b1cb576eba53660a498 (127.9.90.65:40015), f288406707a747eb93376f4e569ea38e (127.9.90.67:34879)
I20260619 10:28:19.228139 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1c575e592947e79942d1b1727eb8ed" candidate_uuid: "d247c2f578814bacabf2c523d5c2896e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f288406707a747eb93376f4e569ea38e"
I20260619 10:28:19.228174 23444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1c575e592947e79942d1b1727eb8ed" candidate_uuid: "d247c2f578814bacabf2c523d5c2896e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df915e826664b1cb576eba53660a498"
I20260619 10:28:19.228216 23712 raft_consensus.cc:3037] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:19.228238 23444 raft_consensus.cc:3037] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:19.228894 23712 raft_consensus.cc:2454] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d247c2f578814bacabf2c523d5c2896e in term 1.
I20260619 10:28:19.228901 23444 raft_consensus.cc:2454] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d247c2f578814bacabf2c523d5c2896e in term 1.
I20260619 10:28:19.229048 23513 leader_election.cc:302] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d247c2f578814bacabf2c523d5c2896e, f288406707a747eb93376f4e569ea38e; no voters:
I20260619 10:28:19.229130 23778 raft_consensus.cc:2781] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:28:19.229262 23778 raft_consensus.cc:686] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 1 LEADER]: Becoming Leader. State: Replica: d247c2f578814bacabf2c523d5c2896e, State: Running, Role: LEADER
I20260619 10:28:19.229346 23778 consensus_queue.cc:237] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.230450 23299 catalog_manager.cc:5495] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e reported cstate change: term changed from 0 to 1, leader changed from <none> to d247c2f578814bacabf2c523d5c2896e (127.9.90.66). New cstate: current_term: 1 leader_uuid: "d247c2f578814bacabf2c523d5c2896e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } health_report { overall_health: UNKNOWN } } }
I20260619 10:28:19.243041 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.240847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.9.90.66:43907)
0619 10:28:19.240884 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.240886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.240894 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:28:19.240924 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.240924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.240949 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.241004 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:28:19.241168 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.241606 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.242373 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.242566 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.242626 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.242737 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:28:19.242826 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:28:19.242880 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.242954 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.242971 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:28:19.244848 23712 raft_consensus.cc:1261] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [term 1 FOLLOWER]: Refusing update from remote peer d247c2f578814bacabf2c523d5c2896e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:19.244848 23444 raft_consensus.cc:1261] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Refusing update from remote peer d247c2f578814bacabf2c523d5c2896e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:19.245188 23782 consensus_queue.cc:1035] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:19.245342 23778 consensus_queue.cc:1035] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:19.247702 23785 mvcc.cc:205] Tried to move back new op lower bound from 7298518627303821312 to 7298518627243974656. Current Snapshot: MvccSnapshot[applied={T|T < 7298518627303821312}]
I20260619 10:28:19.248034 23786 mvcc.cc:205] Tried to move back new op lower bound from 7298518627303821312 to 7298518627243974656. Current Snapshot: MvccSnapshot[applied={T|T < 7298518627303821312 or (T in {7298518627303821312})}]
I20260619 10:28:19.249871 23299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:19.249979 23299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:19.250774 23299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Sending DeleteTablet for 3 replicas of tablet 7e1c575e592947e79942d1b1727eb8ed
I20260619 10:28:19.251127 23424 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1c575e592947e79942d1b1727eb8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:19 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:42656
I20260619 10:28:19.251153 23692 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1c575e592947e79942d1b1727eb8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:19 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:42692
I20260619 10:28:19.251379 23795 tablet_replica.cc:330] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: stopping tablet replica
I20260619 10:28:19.251411 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 23361
I20260619 10:28:19.251504 23795 raft_consensus.cc:2229] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:28:19.251603 23795 raft_consensus.cc:2258] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:19.252092 23795 ts_tablet_manager.cc:1905] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:19.253638 23795 ts_tablet_manager.cc:1918] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:19.253710 23795 log.cc:1196] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/7e1c575e592947e79942d1b1727eb8ed
I20260619 10:28:19.253973 23795 ts_tablet_manager.cc:1939] T 7e1c575e592947e79942d1b1727eb8ed P f288406707a747eb93376f4e569ea38e: Deleting consensus metadata
I20260619 10:28:19.254520 23285 catalog_manager.cc:4841] TS f288406707a747eb93376f4e569ea38e (127.9.90.67:34879): tablet 7e1c575e592947e79942d1b1727eb8ed (table test-table [id=cef599b7375c43058b632f5271a445b8]) successfully deleted
I20260619 10:28:19.254710 23558 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1c575e592947e79942d1b1727eb8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:19 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:50180
I20260619 10:28:19.254915 23796 tablet_replica.cc:330] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: stopping tablet replica
I20260619 10:28:19.255031 23796 raft_consensus.cc:2229] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:28:19.255131 23796 raft_consensus.cc:2258] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:19.255580 23796 ts_tablet_manager.cc:1905] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:19.257133 23796 ts_tablet_manager.cc:1918] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:19.257203 23796 log.cc:1196] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/7e1c575e592947e79942d1b1727eb8ed
I20260619 10:28:19.257457 23796 ts_tablet_manager.cc:1939] T 7e1c575e592947e79942d1b1727eb8ed P d247c2f578814bacabf2c523d5c2896e: Deleting consensus metadata
W20260619 10:28:19.257500 23286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:40015 (error 108)
I20260619 10:28:19.257690 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 23495
I20260619 10:28:19.257838 23284 catalog_manager.cc:4841] TS d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907): tablet 7e1c575e592947e79942d1b1727eb8ed (table test-table [id=cef599b7375c43058b632f5271a445b8]) successfully deleted
I20260619 10:28:19.258585 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.258466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:40015 (local address 127.0.0.1:42670)
0619 10:28:19.258514 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.258554 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 10:28:19.258865 23286 catalog_manager.cc:4584] TS 8df915e826664b1cb576eba53660a498 (127.9.90.65:40015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e1c575e592947e79942d1b1727eb8ed: Network error: Client connection negotiation failed: client connection to 127.9.90.65:40015: connect: Connection refused (error 111)
I20260619 10:28:19.262411 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 23629
I20260619 10:28:19.266026 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 23267
I20260619 10:28:19.270300 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33143
--webserver_interface=127.9.90.126
--webserver_port=34787
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:19.348165 23798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:19.348356 23798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:19.348384 23798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:19.349502 23798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:19.349539 23798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:19.349552 23798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:19.349562 23798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:19.349573 23798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:19.351027 23798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:33143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:19.351220 23798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:19.351397 23798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:19.359280 23805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.359278 23807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:19.359310 23798 server_base.cc:1034] running on GCE node
W20260619 10:28:19.359278 23804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:19.359804 23798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:19.360096 23798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:19.361229 23798 hybrid_clock.cc:648] HybridClock initialized: now 1781864899361220 us; error 31 us; skew 500 ppm
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:19.363559 23798 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:28:19.363983 23798 webserver.cc:460] Webserver started at http://127.9.90.126:34787/ using document root <none> and password file <none>
I20260619 10:28:19.364194 23798 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:19.364256 23798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:19.365556 23798 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:28:19.366134 23814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.366276 23798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.366353 23798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "739bc8f60fd44e729c538f6d920227bc"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "8dc88460aef5b34d9d259f15cfbbe5e4"
server_key_iv: "61c2618f61c3fed401c8efef8127849a"
server_key_version: "encryptionkey@0"
I20260619 10:28:19.366726 23798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:19.382258 23798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:19.384114 23798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:19.384250 23798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:19.388213 23798 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:33143
I20260619 10:28:19.388253 23866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:33143 every 8 connection(s)
I20260619 10:28:19.388648 23798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:28:19.390250 23867 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:28:19.390884 23867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Bootstrap starting.
I20260619 10:28:19.394687 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 23798
I20260619 10:28:19.394937 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:40015
--local_ip_for_outbound_sockets=127.9.90.65
--tserver_master_addrs=127.9.90.126:33143
--webserver_port=39561
--webserver_interface=127.9.90.65
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:19.398640 23867 log.cc:824] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:19.400444 23867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:28:19.400698 23867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Bootstrap complete.
I20260619 10:28:19.402806 23867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:19.403035 23867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739bc8f60fd44e729c538f6d920227bc, State: Initialized, Role: FOLLOWER
I20260619 10:28:19.403191 23867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:19.403263 23867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:28:19.403314 23867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:28:19.403362 23867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:28:19.404320 23867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:19.404412 23867 leader_election.cc:302] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 739bc8f60fd44e729c538f6d920227bc; no voters:
I20260619 10:28:19.404563 23867 leader_election.cc:288] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 10:28:19.404624 23871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:28:19.404757 23871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 2 LEADER]: Becoming Leader. State: Replica: 739bc8f60fd44e729c538f6d920227bc, State: Running, Role: LEADER
I20260619 10:28:19.404819 23867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:19.404857 23871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:19.405110 23871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 739bc8f60fd44e729c538f6d920227bc. Latest consensus state: current_term: 2 leader_uuid: "739bc8f60fd44e729c538f6d920227bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } } }
I20260619 10:28:19.405170 23871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:19.405294 23872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "739bc8f60fd44e729c538f6d920227bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } } }
I20260619 10:28:19.405342 23872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:19.405421 23876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:28:19.406194 23876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:28:19.406287 23876 catalog_manager.cc:1259] Loaded cluster ID: 734d1addbc2a445fa5aa219bf4407205
I20260619 10:28:19.406318 23876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:28:19.406844 23876 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 10:28:19.411454 23876 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 10:28:19.414768 23876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:28:19.414968 23876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Loaded TSK: 0
I20260619 10:28:19.415175 23876 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:28:19.475311 23869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:19.475524 23869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:19.475546 23869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:19.476708 23869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:19.476750 23869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:19.476775 23869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:28:19.478271 23869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:40015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=39561
--tserver_master_addrs=127.9.90.126:33143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:19.478490 23869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:19.478701 23869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:19.484524 23895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.484524 23894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.484598 23897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:19.484735 23869 server_base.cc:1034] running on GCE node
I20260619 10:28:19.484943 23869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:19.485224 23869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:19.486397 23869 hybrid_clock.cc:648] HybridClock initialized: now 1781864899486372 us; error 41 us; skew 500 ppm
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:19.489413 23869 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:28:19.489920 23869 webserver.cc:460] Webserver started at http://127.9.90.65:39561/ using document root <none> and password file <none>
I20260619 10:28:19.490150 23869 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:19.490209 23869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:19.491699 23869 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:19.492370 23904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.492564 23869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:19.492654 23869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "8df915e826664b1cb576eba53660a498"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "269d4f313016fc6037f2267e083d07ca"
server_key_iv: "a3c48282387f5eee594fd51b6de88817"
server_key_version: "encryptionkey@0"
I20260619 10:28:19.493026 23869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:19.522354 23869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:19.524883 23869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:19.525031 23869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:19.525271 23869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:19.525741 23911 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 10:28:19.527199 23869 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 10:28:19.527256 23869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 10:28:19.527283 23869 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 10:28:19.527925 23869 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 10:28:19.527969 23869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 10:28:19.528028 23911 tablet_bootstrap.cc:492] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Bootstrap starting.
I20260619 10:28:19.536372 23869 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:40015
I20260619 10:28:19.536491 24018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:40015 every 8 connection(s)
I20260619 10:28:19.536937 23869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:28:19.538699 23911 log.cc:824] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:19.539748 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 23869
I20260619 10:28:19.540050 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:43907
--local_ip_for_outbound_sockets=127.9.90.66
--tserver_master_addrs=127.9.90.126:33143
--webserver_port=45029
--webserver_interface=127.9.90.66
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:19.540935 23911 tablet_bootstrap.cc:492] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:28:19.541182 23911 tablet_bootstrap.cc:492] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Bootstrap complete.
I20260619 10:28:19.542254 23911 ts_tablet_manager.cc:1397] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.014s
I20260619 10:28:19.543433 23911 raft_consensus.cc:348] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.543741 23911 raft_consensus.cc:729] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df915e826664b1cb576eba53660a498, State: Initialized, Role: FOLLOWER
I20260619 10:28:19.543872 23911 consensus_queue.cc:260] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } } peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } }
I20260619 10:28:19.544124 23911 ts_tablet_manager.cc:1428] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:19.548480 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.537721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:35591)
0619 10:28:19.537887 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.537930 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.538465 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.538892 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.538896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.539247 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:19.539497 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.539504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.541121 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.541122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.542139 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.542142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.542905 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.542906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.543058 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.543610 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:19.543619 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:19.545066 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:19.547074 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.547080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.547081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.547283 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.547547 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.547548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.547548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.547634 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.548059 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:19.548062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:19.548162 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.548233 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.548345 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260619 10:28:19.548642 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.537971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:35591 (local address 127.9.90.126:33143)
0619 10:28:19.538199 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.538207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.538250 (+ 43us) server_negotiation.cc:407] Connection header received
0619 10:28:19.538690 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.538695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.538739 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.538869 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:19.539660 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.540969 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.542228 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.542800 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.542924 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.545165 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:19.545170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:19.545170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:19.545183 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:19.546991 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.547360 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.547363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.547364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.547482 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.547688 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.547689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.547689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.547996 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:19.548083 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.548311 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.548449 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260619 10:28:19.549232 24019 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33143
I20260619 10:28:19.549316 24019 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:19.549476 24019 heartbeater.cc:511] Master 127.9.90.126:33143 requested a full tablet report, sending...
I20260619 10:28:19.550213 23831 ts_manager.cc:194] Registered new tserver with Master: 8df915e826664b1cb576eba53660a498 (127.9.90.65:40015)
I20260619 10:28:19.551419 23831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:35591
I20260619 10:28:19.552260 24019 heartbeater.cc:503] Master 127.9.90.126:33143 was elected leader, sending a full tablet report...
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.65@KRBTEST.COM
I20260619 10:28:19.558478 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.550794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:40015 (local address 127.0.0.1:42680)
0619 10:28:19.550993 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.551048 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.551111 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.551271 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.551275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.551520 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:19.551577 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.551579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.552645 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.552646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.553543 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.553546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.554009 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.554010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.554085 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.554449 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:19.554453 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:19.555774 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:19.557482 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.557487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.557489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.557709 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.557930 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.557931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.557931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.557983 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.558236 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:19.558241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:19.558279 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.558335 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.558416 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260619 10:28:19.558542 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.550864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.9.90.65:40015)
0619 10:28:19.551001 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.551006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.551107 (+ 101us) server_negotiation.cc:407] Connection header received
0619 10:28:19.551196 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.551198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.551211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.551264 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:19.551647 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.552586 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.553624 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.553945 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.553998 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.555865 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:19.555872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:19.555873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:19.555879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:19.557405 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.557775 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.557777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.557778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.557872 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.558033 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.558034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.558034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.558184 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:19.558258 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.558365 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.558451 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260619 10:28:19.558828 23953 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1c575e592947e79942d1b1727eb8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:19 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:42680
I20260619 10:28:19.559315 24028 tablet_replica.cc:330] stopping tablet replica
I20260619 10:28:19.559401 24028 raft_consensus.cc:2229] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:28:19.559459 24028 raft_consensus.cc:2258] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:19.559715 24028 ts_tablet_manager.cc:1905] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:19.561550 24028 ts_tablet_manager.cc:1918] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:19.561607 24028 log.cc:1196] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/7e1c575e592947e79942d1b1727eb8ed
I20260619 10:28:19.561817 24028 ts_tablet_manager.cc:1939] T 7e1c575e592947e79942d1b1727eb8ed P 8df915e826664b1cb576eba53660a498: Deleting consensus metadata
I20260619 10:28:19.562247 23818 catalog_manager.cc:4841] TS 8df915e826664b1cb576eba53660a498 (127.9.90.65:40015): tablet 7e1c575e592947e79942d1b1727eb8ed (table test-table [id=cef599b7375c43058b632f5271a445b8]) successfully deleted
W20260619 10:28:19.623013 24024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:19.623202 24024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:19.623231 24024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:19.624334 24024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:19.624374 24024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:19.624408 24024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:28:19.625862 24024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:43907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=45029
--tserver_master_addrs=127.9.90.126:33143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:19.626084 24024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:19.626291 24024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:19.630910 24034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.630913 24036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.630913 24033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:19.631106 24024 server_base.cc:1034] running on GCE node
I20260619 10:28:19.631368 24024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:19.631598 24024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:19.632737 24024 hybrid_clock.cc:648] HybridClock initialized: now 1781864899632726 us; error 26 us; skew 500 ppm
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:19.635161 24024 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:28:19.635571 24024 webserver.cc:460] Webserver started at http://127.9.90.66:45029/ using document root <none> and password file <none>
I20260619 10:28:19.635782 24024 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:19.635850 24024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:19.637161 24024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:19.637687 24043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.637859 24024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:19.637938 24024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "d247c2f578814bacabf2c523d5c2896e"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "b881393b9821dd933cb15341020db0f3"
server_key_iv: "5e6171ca92246fe25deeefcb64781e20"
server_key_version: "encryptionkey@0"
I20260619 10:28:19.638233 24024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:19.647733 24024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:19.649326 24024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:19.649448 24024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:19.649628 24024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:19.649955 24024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:19.649988 24024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.650015 24024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:19.650040 24024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.655236 24024 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:43907
I20260619 10:28:19.655292 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:43907 every 8 connection(s)
I20260619 10:28:19.655576 24024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:19.664249 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.656235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:41507 (local address 127.9.90.126:33143)
0619 10:28:19.656286 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.656288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.656490 (+ 202us) server_negotiation.cc:407] Connection header received
0619 10:28:19.656977 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.656978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.656993 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.657067 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:19.657928 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.658436 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.659423 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.659633 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.659688 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.661541 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:19.661542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:19.661543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:19.661551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:19.662997 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.663383 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.663384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.663385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.663446 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.663629 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.663629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.663630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.663727 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:19.663776 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.664104 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.664181 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:28:19.664327 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.656213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:41507)
0619 10:28:19.656348 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.656388 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.656868 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.657122 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.657125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.657638 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:19.657850 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.657854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.658514 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.658515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.659308 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.659310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.659694 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.659694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.659782 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.660193 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:19.660201 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:19.661437 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:19.663091 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.663097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.663099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.663303 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.663499 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.663500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.663500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.663584 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.663806 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:19.663811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:19.663940 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.664032 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.664164 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260619 10:28:19.664736 24157 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33143
I20260619 10:28:19.664819 24157 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:19.664975 24157 heartbeater.cc:511] Master 127.9.90.126:33143 requested a full tablet report, sending...
I20260619 10:28:19.665284 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 24024
I20260619 10:28:19.665369 23831 ts_manager.cc:194] Registered new tserver with Master: d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907)
I20260619 10:28:19.665552 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:34879
--local_ip_for_outbound_sockets=127.9.90.67
--tserver_master_addrs=127.9.90.126:33143
--webserver_port=40369
--webserver_interface=127.9.90.67
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:19.665830 23831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:41507
W20260619 10:28:19.742800 24160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:19.742992 24160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:19.743013 24160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:19.744073 24160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:19.744105 24160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:19.744129 24160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:28:19.745452 24160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:34879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=40369
--tserver_master_addrs=127.9.90.126:33143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:19.745615 24160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:19.745780 24160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:19.750332 24165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.750458 24166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.750334 24168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:19.750572 24160 server_base.cc:1034] running on GCE node
I20260619 10:28:19.750751 24160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:19.750945 24160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:19.752082 24160 hybrid_clock.cc:648] HybridClock initialized: now 1781864899752070 us; error 31 us; skew 500 ppm
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:19.754348 24160 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:28:19.754742 24160 webserver.cc:460] Webserver started at http://127.9.90.67:40369/ using document root <none> and password file <none>
I20260619 10:28:19.754922 24160 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:19.754993 24160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:19.756244 24160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:19.756774 24175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.756928 24160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.757014 24160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "f288406707a747eb93376f4e569ea38e"
format_stamp: "Formatted at 2026-06-19 10:28:19 on dist-test-slave-hw7q"
server_key: "2e95bfe5d683972a52af332414c6275e"
server_key_iv: "cc170edac3c1a89aaafcfde6c3eeecce"
server_key_version: "encryptionkey@0"
I20260619 10:28:19.757311 24160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:19.770891 24160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:19.772495 24160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:19.772620 24160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:19.772809 24160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:19.773075 24160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:19.773106 24160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.773132 24160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:19.773149 24160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.778512 24160 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:34879
I20260619 10:28:19.778546 24288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:34879 every 8 connection(s)
I20260619 10:28:19.778863 24160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:28:19.779978 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 24160
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:19.786929 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.780344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.9.90.126:33143)
0619 10:28:19.780449 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.780451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.780543 (+ 92us) server_negotiation.cc:407] Connection header received
0619 10:28:19.780590 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.780590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.780602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.780671 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:19.781183 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.781819 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.782656 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.782983 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.783078 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.784208 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:19.784210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:19.784210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:19.784217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:19.785759 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.786138 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.786140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.786140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.786240 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.786418 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.786420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.786420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.786537 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:19.786698 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.786740 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.786822 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260619 10:28:19.788546 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.779479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:42975 (local address 127.9.90.126:33143)
0619 10:28:19.779532 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.779534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.779769 (+ 235us) server_negotiation.cc:407] Connection header received
0619 10:28:19.780389 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.780391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.780403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.780454 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:19.781543 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.782143 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.783385 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.783563 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.783619 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.786125 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:19.786127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:19.786127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:19.786134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:19.787322 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.787750 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.787751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.787751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.787815 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:19.788034 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:19.788035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:19.788035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:19.788163 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:19.788217 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.788433 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.788491 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:28:19.788714 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.779477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:42975)
0619 10:28:19.779643 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.779676 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.780256 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.780541 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.780545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.781153 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:19.781399 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.781406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.782251 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.782252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.783300 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.783303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.783633 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.783635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.783751 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.784205 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:19.784215 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:19.786052 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:19.787390 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.787397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.787399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.787662 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.787876 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:19.787877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:19.787877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:19.787976 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:19.788217 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:19.788221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:19.788328 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.788415 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.788526 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260619 10:28:19.789108 24289 heartbeater.cc:347] Connected to a master server at 127.9.90.126:33143
I20260619 10:28:19.789189 24289 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:19.789320 24289 heartbeater.cc:511] Master 127.9.90.126:33143 requested a full tablet report, sending...
I20260619 10:28:19.789628 23831 ts_manager.cc:194] Registered new tserver with Master: f288406707a747eb93376f4e569ea38e (127.9.90.67:34879)
I20260619 10:28:19.789981 23831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:42975
I20260619 10:28:19.798866 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:28:19.801263 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.799168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.9.90.126:33143)
0619 10:28:19.799232 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.799234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.799272 (+ 38us) server_negotiation.cc:407] Connection header received
0619 10:28:19.799348 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.799349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.799378 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.799429 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:28:19.799522 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.799969 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.800684 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.800869 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.800919 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.801044 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:28:19.801140 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:28:19.801191 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.801205 (+ 14us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.801219 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 10:28:19.801496 23831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:28:19.803002 23831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:28:19.804924 23953 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb8661d57b14824bddec213d4f8139d (DEFAULT_TABLE table=test-table [id=de33f4db8f614b66aff6b37bc8f467b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:19.805132 23953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb8661d57b14824bddec213d4f8139d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:19.806648 23911 tablet_bootstrap.cc:492] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Bootstrap starting.
I20260619 10:28:19.806967 23911 tablet_bootstrap.cc:654] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:19.807428 23911 tablet_bootstrap.cc:492] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: No bootstrap required, opened a new log
I20260619 10:28:19.807468 23911 ts_tablet_manager.cc:1397] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:28:19.807787 23911 raft_consensus.cc:348] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:19.807871 23911 raft_consensus.cc:374] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:19.807914 23911 raft_consensus.cc:729] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df915e826664b1cb576eba53660a498, State: Initialized, Role: FOLLOWER
I20260619 10:28:19.807960 23911 consensus_queue.cc:260] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:19.808043 23911 ts_tablet_manager.cc:1428] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 10:28:19.808176 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.804525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:34879 (local address 127.0.0.1:42700)
0619 10:28:19.804579 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.804588 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.804617 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.804950 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.804951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.804952 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:19.804991 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.804992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.806115 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.806116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.807018 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.807021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.807947 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.807948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.808016 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.808038 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.808066 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.808111 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 10:28:19.808488 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.804642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.9.90.67:34879)
0619 10:28:19.804773 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.804777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.804787 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:28:19.804823 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.804825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.804839 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.804922 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:19.805114 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.806001 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.807112 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.807852 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.808270 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.808344 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.808369 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.808407 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260619 10:28:19.808833 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.804556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:43907 (local address 127.0.0.1:50188)
0619 10:28:19.804721 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.804731 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.804768 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.805299 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.805300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.805302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:19.805336 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.805337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.806557 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.806558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.807687 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.807690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.808612 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.808613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.808672 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.808675 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.808727 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.808762 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260619 10:28:19.808882 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.804798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.9.90.66:43907)
0619 10:28:19.805050 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.805054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.805103 (+ 49us) server_negotiation.cc:407] Connection header received
0619 10:28:19.805140 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.805143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.805158 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.805229 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:19.805414 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.806444 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.807782 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.808462 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.808524 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.808580 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.808760 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.808807 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":113,"threads_started":1}
I20260619 10:28:19.809139 24223 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb8661d57b14824bddec213d4f8139d (DEFAULT_TABLE table=test-table [id=de33f4db8f614b66aff6b37bc8f467b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:19.809410 24223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb8661d57b14824bddec213d4f8139d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:19.809417 24091 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb8661d57b14824bddec213d4f8139d (DEFAULT_TABLE table=test-table [id=de33f4db8f614b66aff6b37bc8f467b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:19.809644 24091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb8661d57b14824bddec213d4f8139d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:19.812120 24300 tablet_bootstrap.cc:492] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Bootstrap starting.
I20260619 10:28:19.812211 24299 tablet_bootstrap.cc:492] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: Bootstrap starting.
I20260619 10:28:19.812719 24300 tablet_bootstrap.cc:654] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:19.812764 24299 tablet_bootstrap.cc:654] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:19.813154 24300 log.cc:824] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:19.813163 24299 log.cc:824] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:19.813822 24300 tablet_bootstrap.cc:492] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: No bootstrap required, opened a new log
I20260619 10:28:19.813820 24299 tablet_bootstrap.cc:492] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: No bootstrap required, opened a new log
I20260619 10:28:19.813886 24299 ts_tablet_manager.cc:1397] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:28:19.813889 24300 ts_tablet_manager.cc:1397] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:28:19.815212 24299 raft_consensus.cc:348] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:19.815248 24300 raft_consensus.cc:348] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:19.815357 24299 raft_consensus.cc:374] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:19.815377 24300 raft_consensus.cc:374] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:19.815389 24299 raft_consensus.cc:729] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f288406707a747eb93376f4e569ea38e, State: Initialized, Role: FOLLOWER
I20260619 10:28:19.815414 24300 raft_consensus.cc:729] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d247c2f578814bacabf2c523d5c2896e, State: Initialized, Role: FOLLOWER
I20260619 10:28:19.815495 24299 consensus_queue.cc:260] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:19.815519 24300 consensus_queue.cc:260] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:19.815739 24289 heartbeater.cc:503] Master 127.9.90.126:33143 was elected leader, sending a full tablet report...
I20260619 10:28:19.815740 24299 ts_tablet_manager.cc:1428] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:28:19.815757 24157 heartbeater.cc:503] Master 127.9.90.126:33143 was elected leader, sending a full tablet report...
I20260619 10:28:19.815757 24300 ts_tablet_manager.cc:1428] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:28:19.838768 24303 raft_consensus.cc:482] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:19.838871 24303 raft_consensus.cc:504] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:19.839143 24303 leader_election.cc:288] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907), 8df915e826664b1cb576eba53660a498 (127.9.90.65:40015)
I20260619 10:28:19.842093 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.839391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:41715 (local address 127.9.90.66:43907)
0619 10:28:19.839432 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.839434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.839443 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:28:19.839468 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.839469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.839479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.839545 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:19.839716 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.840226 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.841262 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.841773 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.841859 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.841870 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.842001 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.842032 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:28:19.842093 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.839281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:43907 (local address 127.9.90.67:41715)
0619 10:28:19.839322 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.839332 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.839370 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.839563 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.839564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.839566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:19.839635 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.839636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.840297 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.840298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.841189 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.841192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.841858 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.841859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.841940 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.841944 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.841985 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.842023 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":41}
I20260619 10:28:19.842301 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.839345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:58559 (local address 127.9.90.65:40015)
0619 10:28:19.839390 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.839392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.839551 (+ 159us) server_negotiation.cc:407] Connection header received
0619 10:28:19.839587 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.839588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.839614 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.839686 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:19.839856 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.840370 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.841412 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.841850 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.841915 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.841989 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.842189 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.842234 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:28:19.842403 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb8661d57b14824bddec213d4f8139d" candidate_uuid: "f288406707a747eb93376f4e569ea38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247c2f578814bacabf2c523d5c2896e" is_pre_election: true
I20260619 10:28:19.842504 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.839281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:40015 (local address 127.9.90.67:58559)
0619 10:28:19.839483 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.839493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:19.839545 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:19.839716 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:19.839717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:19.839719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:19.839783 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.839784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.840440 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.840440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.841306 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:19.841308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.842022 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.842023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:19.842104 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.842108 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:28:19.842380 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 10:28:19.842423 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":48,"thread_start_us":96,"threads_started":1}
I20260619 10:28:19.842538 24111 raft_consensus.cc:2454] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f288406707a747eb93376f4e569ea38e in term 0.
I20260619 10:28:19.842722 24177 leader_election.cc:302] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d247c2f578814bacabf2c523d5c2896e, f288406707a747eb93376f4e569ea38e; no voters:
I20260619 10:28:19.842744 23973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb8661d57b14824bddec213d4f8139d" candidate_uuid: "f288406707a747eb93376f4e569ea38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df915e826664b1cb576eba53660a498" is_pre_election: true
I20260619 10:28:19.842831 24303 raft_consensus.cc:2781] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:28:19.842885 24303 raft_consensus.cc:482] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:28:19.842880 23973 raft_consensus.cc:2454] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f288406707a747eb93376f4e569ea38e in term 0.
I20260619 10:28:19.842913 24303 raft_consensus.cc:3037] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:19.843551 24303 raft_consensus.cc:504] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:19.843663 24303 leader_election.cc:288] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [CANDIDATE]: Term 1 election: Requested vote from peers d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907), 8df915e826664b1cb576eba53660a498 (127.9.90.65:40015)
I20260619 10:28:19.843789 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb8661d57b14824bddec213d4f8139d" candidate_uuid: "f288406707a747eb93376f4e569ea38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247c2f578814bacabf2c523d5c2896e"
I20260619 10:28:19.843866 24111 raft_consensus.cc:3037] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:19.843842 23973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb8661d57b14824bddec213d4f8139d" candidate_uuid: "f288406707a747eb93376f4e569ea38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df915e826664b1cb576eba53660a498"
I20260619 10:28:19.843922 23973 raft_consensus.cc:3037] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:19.844521 24111 raft_consensus.cc:2454] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f288406707a747eb93376f4e569ea38e in term 1.
I20260619 10:28:19.844569 23973 raft_consensus.cc:2454] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f288406707a747eb93376f4e569ea38e in term 1.
I20260619 10:28:19.844697 24179 leader_election.cc:302] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8df915e826664b1cb576eba53660a498, f288406707a747eb93376f4e569ea38e; no voters:
I20260619 10:28:19.844785 24303 raft_consensus.cc:2781] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:28:19.844933 24303 raft_consensus.cc:686] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 1 LEADER]: Becoming Leader. State: Replica: f288406707a747eb93376f4e569ea38e, State: Running, Role: LEADER
I20260619 10:28:19.845016 24303 consensus_queue.cc:237] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:19.845716 23831 catalog_manager.cc:5495] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e reported cstate change: term changed from 0 to 1, leader changed from <none> to f288406707a747eb93376f4e569ea38e (127.9.90.67). New cstate: current_term: 1 leader_uuid: "f288406707a747eb93376f4e569ea38e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } health_report { overall_health: UNKNOWN } } }
I20260619 10:28:19.855192 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.853234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.9.90.67:34879)
0619 10:28:19.853271 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:19.853273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:19.853279 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:28:19.853330 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:19.853330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:19.853372 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:19.853424 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:28:19.853501 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.853963 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.854643 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:19.854816 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:19.854863 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:19.854956 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:28:19.855037 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:28:19.855088 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:19.855105 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 10:28:19.855119 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 10:28:19.856693 23973 raft_consensus.cc:1261] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Refusing update from remote peer f288406707a747eb93376f4e569ea38e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:19.856693 24111 raft_consensus.cc:1261] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [term 1 FOLLOWER]: Refusing update from remote peer f288406707a747eb93376f4e569ea38e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:19.857033 24306 consensus_queue.cc:1035] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:19.857143 24303 consensus_queue.cc:1035] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:19.859481 24309 mvcc.cc:205] Tried to move back new op lower bound from 7298518629810774016 to 7298518629765832704. Current Snapshot: MvccSnapshot[applied={T|T < 7298518629810774016 or (T in {7298518629810774016})}]
I20260619 10:28:19.859606 24310 mvcc.cc:205] Tried to move back new op lower bound from 7298518629810774016 to 7298518629765832704. Current Snapshot: MvccSnapshot[applied={T|T < 7298518629810774016}]
I20260619 10:28:19.860980 24313 mvcc.cc:205] Tried to move back new op lower bound from 7298518629810774016 to 7298518629765832704. Current Snapshot: MvccSnapshot[applied={T|T < 7298518629810774016}]
I20260619 10:28:19.861654 23831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:19.861745 23831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:19.862414 23831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Sending DeleteTablet for 3 replicas of tablet 6eb8661d57b14824bddec213d4f8139d
I20260619 10:28:19.862694 24223 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eb8661d57b14824bddec213d4f8139d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:19 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:42700
I20260619 10:28:19.862758 24091 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eb8661d57b14824bddec213d4f8139d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:19 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:50188
I20260619 10:28:19.862789 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 23869
I20260619 10:28:19.862923 23953 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eb8661d57b14824bddec213d4f8139d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:19 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:42680
I20260619 10:28:19.865935 24320 tablet_replica.cc:330] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: stopping tablet replica
I20260619 10:28:19.866040 24320 raft_consensus.cc:2229] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:28:19.866146 24320 raft_consensus.cc:2258] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:19.866302 24319 tablet_replica.cc:330] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: stopping tablet replica
I20260619 10:28:19.866420 24319 raft_consensus.cc:2229] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:28:19.866518 24319 raft_consensus.cc:2258] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:19.866832 24320 ts_tablet_manager.cc:1905] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:19.866940 24319 ts_tablet_manager.cc:1905] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 10:28:19.868196 23818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:40015 (error 108)
I20260619 10:28:19.868389 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 24024
I20260619 10:28:19.868992 24319 ts_tablet_manager.cc:1918] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:19.869067 24319 log.cc:1196] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/6eb8661d57b14824bddec213d4f8139d
I20260619 10:28:19.869360 24319 ts_tablet_manager.cc:1939] T 6eb8661d57b14824bddec213d4f8139d P f288406707a747eb93376f4e569ea38e: Deleting consensus metadata
I20260619 10:28:19.869760 23817 catalog_manager.cc:4841] TS f288406707a747eb93376f4e569ea38e (127.9.90.67:34879): tablet 6eb8661d57b14824bddec213d4f8139d (table test-table [id=de33f4db8f614b66aff6b37bc8f467b2]) successfully deleted
I20260619 10:28:19.872803 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 24160
I20260619 10:28:19.873061 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.872940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:43907 (local address 127.0.0.1:50202)
0619 10:28:19.872995 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.873028 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.66:43907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 10:28:19.873160 23816 catalog_manager.cc:4584] TS d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6eb8661d57b14824bddec213d4f8139d: Network error: Client connection negotiation failed: client connection to 127.9.90.66:43907: connect: Connection refused (error 111)
I20260619 10:28:19.874333 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:19.872940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:40015 (local address 127.0.0.1:42692)
0619 10:28:19.874266 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:19.874298 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1308,"mutex_wait_us":1277}
I20260619 10:28:19.876863 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 23798
I20260619 10:28:19.880879 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33143
--webserver_interface=127.9.90.126
--webserver_port=34787
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:19.958411 24322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:19.958597 24322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:19.958618 24322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:19.959707 24322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:19.959744 24322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:19.959757 24322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:19.959767 24322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:19.959779 24322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:19.961191 24322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:33143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:33143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:19.961395 24322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:19.961593 24322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:19.966012 24330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.966019 24327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:19.966019 24328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:19.966220 24322 server_base.cc:1034] running on GCE node
I20260619 10:28:19.966429 24322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:19.966697 24322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:19.967849 24322 hybrid_clock.cc:648] HybridClock initialized: now 1781864899967843 us; error 45 us; skew 500 ppm
Jun 19 10:28:19 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:19.973503 24322 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:28:19.973996 24322 webserver.cc:460] Webserver started at http://127.9.90.126:34787/ using document root <none> and password file <none>
I20260619 10:28:19.974201 24322 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:19.974253 24322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:19.975610 24322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:19.976276 24337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:19.976425 24322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:19.976513 24322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "739bc8f60fd44e729c538f6d920227bc"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "8dc88460aef5b34d9d259f15cfbbe5e4"
server_key_iv: "61c2618f61c3fed401c8efef8127849a"
server_key_version: "encryptionkey@0"
I20260619 10:28:19.976871 24322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:20.000540 24322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:20.002614 24322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:20.002743 24322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:20.006815 24322 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:33143
I20260619 10:28:20.006896 24389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:33143 every 8 connection(s)
I20260619 10:28:20.007323 24322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:28:20.009341 24390 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:28:20.010521 24390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Bootstrap starting.
I20260619 10:28:20.015318 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 24322
I20260619 10:28:20.015648 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:40015
--local_ip_for_outbound_sockets=127.9.90.65
--tserver_master_addrs=127.9.90.126:33143
--webserver_port=39561
--webserver_interface=127.9.90.65
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:20.021961 24390 log.cc:824] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:20.024621 24390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:28:20.024924 24390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc: Bootstrap complete.
I20260619 10:28:20.027413 24390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:20.027738 24390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739bc8f60fd44e729c538f6d920227bc, State: Initialized, Role: FOLLOWER
I20260619 10:28:20.027913 24390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:20.027983 24390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:28:20.028029 24390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:28:20.028069 24390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 2 FOLLOWER]: Advancing to term 3
I20260619 10:28:20.029381 24390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:20.029490 24390 leader_election.cc:302] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 739bc8f60fd44e729c538f6d920227bc; no voters:
I20260619 10:28:20.029664 24390 leader_election.cc:288] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 10:28:20.029743 24394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 3 FOLLOWER]: Leader election won for term 3
I20260619 10:28:20.030236 24394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [term 3 LEADER]: Becoming Leader. State: Replica: 739bc8f60fd44e729c538f6d920227bc, State: Running, Role: LEADER
I20260619 10:28:20.030383 24394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } }
I20260619 10:28:20.029943 24390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:20.030782 24395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "739bc8f60fd44e729c538f6d920227bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } } }
I20260619 10:28:20.030884 24395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:20.031080 24398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:28:20.030788 24396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 739bc8f60fd44e729c538f6d920227bc. Latest consensus state: current_term: 3 leader_uuid: "739bc8f60fd44e729c538f6d920227bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739bc8f60fd44e729c538f6d920227bc" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 33143 } } }
I20260619 10:28:20.031286 24396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739bc8f60fd44e729c538f6d920227bc [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:20.031868 24398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:28:20.031970 24398 catalog_manager.cc:1259] Loaded cluster ID: 734d1addbc2a445fa5aa219bf4407205
I20260619 10:28:20.032002 24398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 10:28:20.032480 24398 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff2cd7109ad google::LogMessage::Fail() at ??:0
@ 0x7ff2cd71388a google::LogMessage::SendToLog() at ??:0
@ 0x7ff2cd7104de google::LogMessage::Flush() at ??:0
@ 0x7ff2cd711a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff2d56310f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff2d563a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff2cf5de0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff2cf5d3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff2d04376db start_thread at ??:0
@ 0x7ff2cc20e61f clone at ??:0
W20260619 10:28:20.113605 24392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:20.113831 24392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:20.113859 24392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:20.115026 24392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:20.115067 24392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:20.115093 24392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:28:20.116529 24392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:40015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=39561
--tserver_master_addrs=127.9.90.126:33143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:20.116750 24392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:20.116966 24392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:20.121701 24429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:20.121727 24427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:20.121857 24392 server_base.cc:1034] running on GCE node
W20260619 10:28:20.121899 24426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:20.122191 24392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:20.122429 24392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:20.123566 24392 hybrid_clock.cc:648] HybridClock initialized: now 1781864900123552 us; error 31 us; skew 500 ppm
Jun 19 10:28:20 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:20.126031 24392 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:28:20.126487 24392 webserver.cc:460] Webserver started at http://127.9.90.65:39561/ using document root <none> and password file <none>
I20260619 10:28:20.126699 24392 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:20.126751 24392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:20.128103 24392 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:20.128827 24436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:20.128988 24392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:20.129077 24392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "8df915e826664b1cb576eba53660a498"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "269d4f313016fc6037f2267e083d07ca"
server_key_iv: "a3c48282387f5eee594fd51b6de88817"
server_key_version: "encryptionkey@0"
I20260619 10:28:20.129429 24392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:20.142946 24392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:20.144965 24392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:20.145114 24392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:20.145337 24392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:20.145761 24443 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 10:28:20.147596 24392 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 10:28:20.147652 24392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 10:28:20.147689 24392 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 10:28:20.148299 24392 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 10:28:20.148373 24392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 10:28:20.148427 24443 tablet_bootstrap.cc:492] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Bootstrap starting.
I20260619 10:28:20.155511 24392 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:40015
I20260619 10:28:20.155665 24550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:40015 every 8 connection(s)
I20260619 10:28:20.155922 24392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:28:20.158710 24443 log.cc:824] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:20.161276 24443 tablet_bootstrap.cc:492] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:28:20.161569 24443 tablet_bootstrap.cc:492] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Bootstrap complete.
I20260619 10:28:20.161890 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 24392
I20260619 10:28:20.162134 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:43907
--local_ip_for_outbound_sockets=127.9.90.66
--tserver_master_addrs=127.9.90.126:33143
--webserver_port=45029
--webserver_interface=127.9.90.66
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:20.162612 24443 ts_tablet_manager.cc:1397] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260619 10:28:20.163980 24443 raft_consensus.cc:348] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:20.164299 24443 raft_consensus.cc:729] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df915e826664b1cb576eba53660a498, State: Initialized, Role: FOLLOWER
I20260619 10:28:20.164462 24443 consensus_queue.cc:260] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:20.164729 24443 ts_tablet_manager.cc:1428] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260619 10:28:20.207705 24553 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:28:20.156533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:45803)
0619 10:28:20.156695 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:20.156737 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:20.157173 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:20.207499 (+ 50326us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
W20260619 10:28:20.207989 24551 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:33143 (0 consecutive failures): Network error: Failed to ping master at 127.9.90.126:33143: Client connection negotiation failed: client connection to 127.9.90.126:33143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:28:20.208313 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:20.208198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:46493)
0619 10:28:20.208242 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:20.208289 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 10:28:20.208668 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:20.208585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:39863)
0619 10:28:20.208626 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:20.208651 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 10:28:20.208833 24551 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:28:20.241701 24555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:20.241938 24555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:20.241964 24555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:20.243070 24555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:20.243108 24555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:20.243132 24555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:28:20.244493 24555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:43907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=45029
--tserver_master_addrs=127.9.90.126:33143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:20.244679 24555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:20.244889 24555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:20.249506 24561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:20.249517 24564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:20.249532 24555 server_base.cc:1034] running on GCE node
W20260619 10:28:20.249519 24562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:20.249853 24555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:20.250044 24555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:20.251179 24555 hybrid_clock.cc:648] HybridClock initialized: now 1781864900251167 us; error 28 us; skew 500 ppm
Jun 19 10:28:20 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:20.253352 24555 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:28:20.253700 24555 webserver.cc:460] Webserver started at http://127.9.90.66:45029/ using document root <none> and password file <none>
I20260619 10:28:20.253896 24555 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:20.253939 24555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:20.255090 24555 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:20.255780 24571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:20.255975 24555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:20.256059 24555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "d247c2f578814bacabf2c523d5c2896e"
format_stamp: "Formatted at 2026-06-19 10:28:18 on dist-test-slave-hw7q"
server_key: "b881393b9821dd933cb15341020db0f3"
server_key_iv: "5e6171ca92246fe25deeefcb64781e20"
server_key_version: "encryptionkey@0"
I20260619 10:28:20.256356 24555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:20.274166 24555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:20.275827 24555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:20.275947 24555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:20.276134 24555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:20.276502 24578 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 10:28:20.277779 24578 ts_tablet_manager.cc:1852] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 10:28:20.278066 24578 ts_tablet_manager.cc:1905] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:20.279793 24578 ts_tablet_manager.cc:1918] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:20.279870 24578 log.cc:1196] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/6eb8661d57b14824bddec213d4f8139d
I20260619 10:28:20.280057 24578 ts_tablet_manager.cc:1939] T 6eb8661d57b14824bddec213d4f8139d P d247c2f578814bacabf2c523d5c2896e: Deleting consensus metadata
I20260619 10:28:20.280186 24555 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 10:28:20.280231 24555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 10:28:20.280268 24555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:20.280292 24555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:20.285677 24555 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:43907
I20260619 10:28:20.285738 24685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:43907 every 8 connection(s)
I20260619 10:28:20.286010 24555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:28:20.286918 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 24555
I20260619 10:28:20.287009 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:20.286635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:37989)
0619 10:28:20.286798 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:20.286872 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260619 10:28:20.287171 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:34879
--local_ip_for_outbound_sockets=127.9.90.67
--tserver_master_addrs=127.9.90.126:33143
--webserver_port=40369
--webserver_interface=127.9.90.67
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:20.287189 24686 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:33143 (0 consecutive failures): Network error: Failed to ping master at 127.9.90.126:33143: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
I20260619 10:28:20.287552 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:20.287450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:45749)
0619 10:28:20.287497 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:20.287529 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 10:28:20.287871 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:20.287792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:53133)
0619 10:28:20.287830 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:20.287852 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 10:28:20.287950 24686 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:28:20.365120 24689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:20.365316 24689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:20.365338 24689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:20.366488 24689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:20.366528 24689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:20.366554 24689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:28:20.367939 24689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:34879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=40369
--tserver_master_addrs=127.9.90.126:33143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:20.368145 24689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:20.368343 24689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:20.373075 24695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:20.373075 24697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:20.373118 24694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:20.373358 24689 server_base.cc:1034] running on GCE node
I20260619 10:28:20.373555 24689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:20.373780 24689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:20.374948 24689 hybrid_clock.cc:648] HybridClock initialized: now 1781864900374928 us; error 35 us; skew 500 ppm
Jun 19 10:28:20 dist-test-slave-hw7q krb5kdc[23254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:20.377193 24689 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:28:20.377554 24689 webserver.cc:460] Webserver started at http://127.9.90.67:40369/ using document root <none> and password file <none>
I20260619 10:28:20.377760 24689 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:20.377839 24689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:20.379098 24689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:28:20.379695 24704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:20.379886 24689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:28:20.379969 24689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "f288406707a747eb93376f4e569ea38e"
format_stamp: "Formatted at 2026-06-19 10:28:19 on dist-test-slave-hw7q"
server_key: "2e95bfe5d683972a52af332414c6275e"
server_key_iv: "cc170edac3c1a89aaafcfde6c3eeecce"
server_key_version: "encryptionkey@0"
I20260619 10:28:20.380302 24689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:20.389430 24689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:20.391286 24689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:20.391409 24689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:20.391592 24689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:20.391841 24689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:20.391871 24689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:20.391896 24689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:20.391914 24689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:20.397100 24689 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:34879
I20260619 10:28:20.397156 24817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:34879 every 8 connection(s)
I20260619 10:28:20.397441 24689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:28:20.398342 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:20.398019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:39471)
0619 10:28:20.398151 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:20.398211 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
W20260619 10:28:20.398697 24818 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:33143 (0 consecutive failures): Network error: Failed to ping master at 127.9.90.126:33143: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
I20260619 10:28:20.398936 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:20.398850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:38471)
0619 10:28:20.398891 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:20.398918 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 10:28:20.399163 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:20.399098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:44587)
0619 10:28:20.399130 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:20.399149 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260619 10:28:20.399252 24818 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 10:28:20.401690 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 24689
I20260619 10:28:21.209434 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:21.209196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:50033)
0619 10:28:21.209339 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:21.209377 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260619 10:28:21.288479 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:21.288270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:38383)
0619 10:28:21.288378 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:21.288412 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260619 10:28:21.399757 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:21.399540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:46857)
0619 10:28:21.399670 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:21.399705 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260619 10:28:21.413722 24825 raft_consensus.cc:482] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:21.413913 24825 raft_consensus.cc:504] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:21.414213 24825 leader_election.cc:288] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f288406707a747eb93376f4e569ea38e (127.9.90.67:34879), d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907)
Jun 19 10:28:21 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.67@KRBTEST.COM
Jun 19 10:28:21 dist-test-slave-hw7q krb5kdc[23254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.66@KRBTEST.COM
I20260619 10:28:21.425827 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:21.414507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:38599 (local address 127.9.90.66:43907)
0619 10:28:21.414635 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:21.414643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:21.414658 (+ 15us) server_negotiation.cc:407] Connection header received
0619 10:28:21.415265 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:21.415269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:21.415311 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:21.415427 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:21.416399 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:21.417770 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:21.419284 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:21.419919 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:21.420030 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:21.422885 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:21.422889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:21.422889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:21.422899 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:21.424326 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:21.424699 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:21.424702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:21.424703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:21.424779 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:21.424970 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:21.424971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:21.424971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:21.425188 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:21.425255 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:21.425576 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 10:28:21.425689 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":50,"threads_started":1}
I20260619 10:28:21.425810 24827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:21.414423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:43907 (local address 127.9.90.65:38599)
0619 10:28:21.414535 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:21.414543 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:21.414580 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:21.415463 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:21.415466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:21.415986 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:21.416247 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:21.416257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:21.417918 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:21.417918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:21.419179 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:21.419182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:21.420024 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:21.420026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:21.420186 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:21.420746 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:21.420756 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:21.422785 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:21.424436 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:21.424437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:21.424437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:21.424633 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:21.424835 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:21.424836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:21.424836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:21.424916 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:21.425284 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:21.425285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:21.425411 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 10:28:21.425511 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 10:28:21.425690 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":68,"thread_start_us":60,"threads_started":1}
I20260619 10:28:21.425834 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:21.414426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:53995 (local address 127.9.90.67:34879)
0619 10:28:21.414574 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:21.414587 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:21.414622 (+ 35us) server_negotiation.cc:407] Connection header received
0619 10:28:21.415177 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:21.415181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:21.415241 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:21.415427 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:21.416401 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:21.417821 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:21.419284 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:21.419912 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:21.420031 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:21.422627 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:21.422631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:21.422632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:21.422641 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:21.424262 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:21.424662 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:21.424665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:21.424665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:21.424748 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:21.424966 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:21.424967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:21.424967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:21.425208 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:21.425281 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:21.425573 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 10:28:21.425690 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":45,"threads_started":1}
I20260619 10:28:21.425901 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:21.414347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:34879 (local address 127.9.90.65:53995)
0619 10:28:21.414397 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:21.414409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:21.414469 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:21.415463 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:21.415466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:21.415986 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:21.416247 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:21.416257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:21.417902 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:21.417903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:21.419179 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:21.419182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:21.420026 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:21.420027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:21.420186 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:21.420767 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:21.420771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:21.422520 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:21.424339 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:21.424345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:21.424347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:21.424592 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:21.424805 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:21.424806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:21.424806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:21.424906 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:21.425264 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:21.425270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:21.425410 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 10:28:21.425510 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 10:28:21.425765 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":172}
W20260619 10:28:21.426647 24439 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f288406707a747eb93376f4e569ea38e (127.9.90.67:34879): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
W20260619 10:28:21.426715 24438 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
I20260619 10:28:21.426800 24438 leader_election.cc:302] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df915e826664b1cb576eba53660a498; no voters: d247c2f578814bacabf2c523d5c2896e, f288406707a747eb93376f4e569ea38e
I20260619 10:28:21.426903 24825 raft_consensus.cc:2726] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:28:22.210108 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:22.209907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:33811)
0619 10:28:22.210031 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:22.210065 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260619 10:28:22.289108 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:22.288916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:49337)
0619 10:28:22.289035 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:22.289071 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260619 10:28:22.400306 24831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:22.400111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:58623)
0619 10:28:22.400236 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:22.400269 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260619 10:28:23.210741 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:23.210547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:34805)
0619 10:28:23.210661 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:23.210696 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260619 10:28:23.289701 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:23.289520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:47773)
0619 10:28:23.289627 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:23.289661 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260619 10:28:23.370620 24834 raft_consensus.cc:482] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:23.370720 24834 raft_consensus.cc:504] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:23.370886 24834 leader_election.cc:288] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f288406707a747eb93376f4e569ea38e (127.9.90.67:34879), d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907)
W20260619 10:28:23.371304 24439 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f288406707a747eb93376f4e569ea38e (127.9.90.67:34879): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
W20260619 10:28:23.371428 24438 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
I20260619 10:28:23.371472 24438 leader_election.cc:302] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df915e826664b1cb576eba53660a498; no voters: d247c2f578814bacabf2c523d5c2896e, f288406707a747eb93376f4e569ea38e
I20260619 10:28:23.371549 24834 raft_consensus.cc:2726] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:28:23.400867 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:23.400686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:43417)
0619 10:28:23.400789 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:23.400818 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260619 10:28:24.211385 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:24.211152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:59871)
0619 10:28:24.211299 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:24.211337 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260619 10:28:24.290341 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:24.290157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:58523)
0619 10:28:24.290266 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:24.290302 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260619 10:28:24.401495 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:24.401303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:38579)
0619 10:28:24.401417 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:24.401451 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260619 10:28:25.060876 24839 raft_consensus.cc:482] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:25.060977 24839 raft_consensus.cc:504] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:25.061133 24839 leader_election.cc:288] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f288406707a747eb93376f4e569ea38e (127.9.90.67:34879), d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907)
W20260619 10:28:25.061498 24438 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
W20260619 10:28:25.061586 24439 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f288406707a747eb93376f4e569ea38e (127.9.90.67:34879): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
I20260619 10:28:25.061645 24439 leader_election.cc:302] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df915e826664b1cb576eba53660a498; no voters: d247c2f578814bacabf2c523d5c2896e, f288406707a747eb93376f4e569ea38e
I20260619 10:28:25.061736 24839 raft_consensus.cc:2726] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:28:25.212105 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:25.211884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:59837)
0619 10:28:25.212028 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:25.212059 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260619 10:28:25.290941 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:25.290750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:36847)
0619 10:28:25.290871 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:25.290906 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
I20260619 10:28:25.402150 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:25.401913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:42069)
0619 10:28:25.402072 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:25.402109 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260619 10:28:26.212827 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:26.212603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:55691)
0619 10:28:26.212744 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:26.212782 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260619 10:28:26.291503 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:26.291312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:51161)
0619 10:28:26.291435 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:26.291466 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260619 10:28:26.402760 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:26.402570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:55669)
0619 10:28:26.402680 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:26.402717 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260619 10:28:27.213503 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:27.213245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:53059)
0619 10:28:27.213390 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:27.213456 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260619 10:28:27.292127 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:27.291859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:58295)
0619 10:28:27.292033 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:27.292075 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260619 10:28:27.403426 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:27.403212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:44325)
0619 10:28:27.403337 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:27.403378 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260619 10:28:28.214247 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:28.214048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:44485)
0619 10:28:28.214167 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:28.214202 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260619 10:28:28.292766 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:28.292550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:51401)
0619 10:28:28.292685 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:28.292723 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260619 10:28:28.404058 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:28.403865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:44431)
0619 10:28:28.403975 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:28.404016 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260619 10:28:28.714666 24852 raft_consensus.cc:482] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:28.714780 24852 raft_consensus.cc:504] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:28.714962 24852 leader_election.cc:288] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f288406707a747eb93376f4e569ea38e (127.9.90.67:34879), d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907)
W20260619 10:28:28.715294 24439 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f288406707a747eb93376f4e569ea38e (127.9.90.67:34879): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
W20260619 10:28:28.715430 24438 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
I20260619 10:28:28.715497 24438 leader_election.cc:302] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df915e826664b1cb576eba53660a498; no voters: d247c2f578814bacabf2c523d5c2896e, f288406707a747eb93376f4e569ea38e
I20260619 10:28:28.715576 24852 raft_consensus.cc:2726] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:28:29.214891 24853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:29.214670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:46877)
0619 10:28:29.214805 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:29.214845 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260619 10:28:29.293383 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:29.293182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:47081)
0619 10:28:29.293302 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:29.293343 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260619 10:28:29.404752 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:29.404514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:41619)
0619 10:28:29.404658 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:29.404708 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260619 10:28:30.215559 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:30.215347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:59339)
0619 10:28:30.215467 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:30.215514 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260619 10:28:30.294090 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:30.293869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:50501)
0619 10:28:30.294000 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:30.294047 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260619 10:28:30.405361 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:30.405181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:45897)
0619 10:28:30.405286 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:30.405318 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260619 10:28:31.216192 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:31.215993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:52089)
0619 10:28:31.216108 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:31.216144 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260619 10:28:31.294687 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:31.294485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:32857)
0619 10:28:31.294609 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:31.294643 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260619 10:28:31.406018 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:31.405774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:50133)
0619 10:28:31.405934 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:31.405974 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260619 10:28:32.216909 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:32.216695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:51195)
0619 10:28:32.216825 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:32.216859 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260619 10:28:32.295361 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:32.295132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:35325)
0619 10:28:32.295276 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:32.295316 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260619 10:28:32.406860 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:32.406592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:55595)
0619 10:28:32.406757 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:32.406804 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260619 10:28:33.217608 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:33.217359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:55951)
0619 10:28:33.217526 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:33.217563 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260619 10:28:33.296023 24866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:33.295805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:47179)
0619 10:28:33.295942 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:33.295979 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260619 10:28:33.407409 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:33.407225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:44879)
0619 10:28:33.407336 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:33.407369 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260619 10:28:33.889909 24868 raft_consensus.cc:482] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:33.890002 24868 raft_consensus.cc:504] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f288406707a747eb93376f4e569ea38e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34879 } } peers { permanent_uuid: "d247c2f578814bacabf2c523d5c2896e" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 43907 } } peers { permanent_uuid: "8df915e826664b1cb576eba53660a498" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 40015 } }
I20260619 10:28:33.890170 24868 leader_election.cc:288] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f288406707a747eb93376f4e569ea38e (127.9.90.67:34879), d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907)
W20260619 10:28:33.890527 24439 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f288406707a747eb93376f4e569ea38e (127.9.90.67:34879): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
W20260619 10:28:33.890620 24438 leader_election.cc:341] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d247c2f578814bacabf2c523d5c2896e (127.9.90.66:43907): Not found: Tablet not found: 6eb8661d57b14824bddec213d4f8139d
I20260619 10:28:33.890661 24438 leader_election.cc:302] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df915e826664b1cb576eba53660a498; no voters: d247c2f578814bacabf2c523d5c2896e, f288406707a747eb93376f4e569ea38e
I20260619 10:28:33.890734 24868 raft_consensus.cc:2726] T 6eb8661d57b14824bddec213d4f8139d P 8df915e826664b1cb576eba53660a498 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:28:34.218240 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:34.218028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:46867)
0619 10:28:34.218156 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:34.218192 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260619 10:28:34.296710 24870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:34.296491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:36479)
0619 10:28:34.296627 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:34.296665 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260619 10:28:34.408001 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:34.407799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:52681)
0619 10:28:34.407911 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:34.407944 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260619 10:28:35.218958 24872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:35.218742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.65:52963)
0619 10:28:35.218877 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:35.218913 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260619 10:28:35.297441 24873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:35.297196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.66:35075)
0619 10:28:35.297365 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:35.297399 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260619 10:28:35.408725 24874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:35.408505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:33143 (local address 127.9.90.67:53731)
0619 10:28:35.408642 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:35.408679 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260619 10:28:35.410471 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 24392
I20260619 10:28:35.414520 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 24555
I20260619 10:28:35.417902 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 24689
2026-06-19T10:28:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17978 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:28:35 dist-test-slave-hw7q krb5kdc[24877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:28:35 dist-test-slave-hw7q krb5kdc[24877](info): set up 2 sockets
Jun 19 10:28:35 dist-test-slave-hw7q krb5kdc[24877](info): commencing operation
krb5kdc: starting...
W20260619 10:28:36.365635 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:28:36 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:28:36.474754 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36721
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:36721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:36.552090 24890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:36.552284 24890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:36.552304 24890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:36.553388 24890 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 10:28:36.553431 24890 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 10:28:36.553442 24890 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 10:28:36.553452 24890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:36.553463 24890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:36.553474 24890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:36.553484 24890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:36.553499 24890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:36.554910 24890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:36721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:36.555097 24890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:36.555258 24890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:36.559888 24896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:36.559965 24890 server_base.cc:1034] running on GCE node
W20260619 10:28:36.559906 24895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:36.559863 24898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:36.560318 24890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:36.560622 24890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:36.561805 24890 hybrid_clock.cc:648] HybridClock initialized: now 1781864916561773 us; error 32 us; skew 500 ppm
Jun 19 10:28:36 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864916, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:36.564172 24890 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:28:36.564549 24890 webserver.cc:460] Webserver started at http://127.9.90.126:33515/ using document root <none> and password file <none>
I20260619 10:28:36.564718 24890 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:36.564756 24890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:36.564845 24890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:36.565822 24890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "b2d6d71cc0cc4fbdaee619f58138d65d"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "7e4b946570cfcb2e0c1e29403a9981c8"
server_key_iv: "f430dfed7ed006be2fef8dd18132cc68"
server_key_version: "encryptionkey@0"
I20260619 10:28:36.566120 24890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "b2d6d71cc0cc4fbdaee619f58138d65d"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "7e4b946570cfcb2e0c1e29403a9981c8"
server_key_iv: "f430dfed7ed006be2fef8dd18132cc68"
server_key_version: "encryptionkey@0"
I20260619 10:28:36.567296 24890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:28:36.567955 24905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:36.568094 24890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:28:36.568172 24890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "b2d6d71cc0cc4fbdaee619f58138d65d"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "7e4b946570cfcb2e0c1e29403a9981c8"
server_key_iv: "f430dfed7ed006be2fef8dd18132cc68"
server_key_version: "encryptionkey@0"
I20260619 10:28:36.568224 24890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:36.577946 24890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:36.579566 24890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:36.579661 24890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:36.583141 24890 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:36721
I20260619 10:28:36.583170 24957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:36721 every 8 connection(s)
I20260619 10:28:36.583463 24890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:28:36.584219 24958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:36.586424 24958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Bootstrap starting.
I20260619 10:28:36.586961 24958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:36.587220 24958 log.cc:824] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:36.587822 24958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: No bootstrap required, opened a new log
I20260619 10:28:36.589136 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 24890
I20260619 10:28:36.589232 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:28:36.589432 9577 external_mini_cluster.cc:1428] Setting key 5461be4f5ae5e1042634036a10b3abe2
I20260619 10:28:36.589599 24958 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:28:36.589736 24958 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:36.589767 24958 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d6d71cc0cc4fbdaee619f58138d65d, State: Initialized, Role: FOLLOWER
I20260619 10:28:36.589975 24958 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:28:36.590090 24958 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:28:36.590135 24958 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:28:36.590173 24958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:36.590917 24958 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:28:36.591008 24958 leader_election.cc:302] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d6d71cc0cc4fbdaee619f58138d65d; no voters:
I20260619 10:28:36.591152 24958 leader_election.cc:288] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:28:36.591190 24963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:28:36.591311 24963 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 1 LEADER]: Becoming Leader. State: Replica: b2d6d71cc0cc4fbdaee619f58138d65d, State: Running, Role: LEADER
I20260619 10:28:36.591426 24958 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:36.591437 24963 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:28:36.591778 24963 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d6d71cc0cc4fbdaee619f58138d65d. Latest consensus state: current_term: 1 leader_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } } }
I20260619 10:28:36.592023 24963 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:36.592392 24972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:28:36.591948 24964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } } }
I20260619 10:28:36.592787 24964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:36.592911 24972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:28:36.594900 24972 catalog_manager.cc:1347] Generated new cluster ID: 99e0305724d2431bad8a7172d27c4956
I20260619 10:28:36.594952 24972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:28:36 dist-test-slave-hw7q krb5kdc[24877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:36.599366 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:36.589910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.9.90.126:36721)
0619 10:28:36.590081 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:36.590090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:36.590122 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:28:36.590237 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:36.590244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:36.590316 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:36.590433 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:36.591007 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.592421 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.593437 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.593990 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.594091 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:36.596239 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:36.596243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:36.596244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:36.596255 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:36.597955 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:36.598286 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:36.598288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:36.598289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:36.598391 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:36.598579 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:36.598580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:36.598580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:36.598815 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:36.598952 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:36.599071 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 10:28:36.599201 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260619 10:28:36.634225 24972 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:28:36.634974 24972 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:28:36.644266 24972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Generated new TSK 0
I20260619 10:28:36.644472 24972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:28:36.670485 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36721
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:36.769213 24984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:36.769414 24984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:36.769445 24984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:36.770573 24984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:36.770618 24984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:36.770651 24984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:28:36.772058 24984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:36.772267 24984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:36.772462 24984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:36.777061 24989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:36.777061 24992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:36.777194 24984 server_base.cc:1034] running on GCE node
W20260619 10:28:36.777278 24990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:36.777513 24984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:36.777735 24984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:36.778882 24984 hybrid_clock.cc:648] HybridClock initialized: now 1781864916778864 us; error 38 us; skew 500 ppm
Jun 19 10:28:36 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864916, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:36.781281 24984 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:28:36.781673 24984 webserver.cc:460] Webserver started at http://127.9.90.65:46057/ using document root <none> and password file <none>
I20260619 10:28:36.781894 24984 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:36.781940 24984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:36.782061 24984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:36.783052 24984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "ad281fd2bdae498180ad14ee0f9a8a70"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "c00d52d4441aced37465fca6ecd99b7b"
server_key_iv: "e9cd184224bd228b302c4052c15c97b0"
server_key_version: "encryptionkey@0"
I20260619 10:28:36.783357 24984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "ad281fd2bdae498180ad14ee0f9a8a70"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "c00d52d4441aced37465fca6ecd99b7b"
server_key_iv: "e9cd184224bd228b302c4052c15c97b0"
server_key_version: "encryptionkey@0"
I20260619 10:28:36.784528 24984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:28:36.785152 24999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:36.785303 24984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:36.785373 24984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "ad281fd2bdae498180ad14ee0f9a8a70"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "c00d52d4441aced37465fca6ecd99b7b"
server_key_iv: "e9cd184224bd228b302c4052c15c97b0"
server_key_version: "encryptionkey@0"
I20260619 10:28:36.785434 24984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:36.802682 24984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:36.804311 24984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:36.804422 24984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:36.804602 24984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:36.804869 24984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:36.804900 24984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:36.804920 24984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:36.804942 24984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:36.810248 24984 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:34201
I20260619 10:28:36.810300 25112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:34201 every 8 connection(s)
I20260619 10:28:36.810575 24984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:28:36.815256 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 24984
I20260619 10:28:36.815363 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:28:36.815578 9577 external_mini_cluster.cc:1428] Setting key ea2778fe6e30e4f95e4fd68cc6f3b151
Jun 19 10:28:36 dist-test-slave-hw7q krb5kdc[24877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864916, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:36.819275 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:36.811205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:51409 (local address 127.9.90.126:36721)
0619 10:28:36.811253 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:36.811255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:36.811469 (+ 214us) server_negotiation.cc:407] Connection header received
0619 10:28:36.811960 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:36.811961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:36.811974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:36.812031 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:36.812981 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.813503 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.814478 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.814677 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.814738 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:36.816812 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:36.816813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:36.816814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:36.816821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:36.818115 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:36.818504 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:36.818505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:36.818505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:36.818548 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:36.818772 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:36.818773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:36.818773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:36.818860 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:36.818902 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:36.819159 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 10:28:36.819215 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:28:36.819509 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:36.811184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:51409)
0619 10:28:36.811362 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:36.811398 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:36.811848 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:36.812138 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:36.812142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:36.812714 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:36.812894 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:36.812899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.813571 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.813572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:36.814397 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:36.814399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.814777 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.814778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:36.814882 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:36.815304 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:36.815313 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:36.816723 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:36.818186 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:36.818192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:36.818194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:36.818442 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:36.818598 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:36.818599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:36.818599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:36.818717 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:36.818907 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:36.818911 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:36.819050 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 10:28:36.819150 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 10:28:36.819300 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260619 10:28:36.819958 25113 heartbeater.cc:347] Connected to a master server at 127.9.90.126:36721
I20260619 10:28:36.820039 25113 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:36.820189 25113 heartbeater.cc:511] Master 127.9.90.126:36721 requested a full tablet report, sending...
I20260619 10:28:36.820645 24922 ts_manager.cc:194] Registered new tserver with Master: ad281fd2bdae498180ad14ee0f9a8a70 (127.9.90.65:34201)
I20260619 10:28:36.821211 24922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:51409
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:28:36.838573 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36721
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:36.915213 25118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:36.915402 25118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:36.915422 25118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:36.916507 25118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:36.916545 25118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:36.916569 25118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:28:36.917970 25118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:36.918175 25118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:36.918368 25118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:36.922963 25123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:36.922978 25124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:36.922986 25126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:36.923050 25118 server_base.cc:1034] running on GCE node
I20260619 10:28:36.923380 25118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:36.923624 25118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:36.924784 25118 hybrid_clock.cc:648] HybridClock initialized: now 1781864916924755 us; error 39 us; skew 500 ppm
Jun 19 10:28:36 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864916, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:36.927136 25118 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:28:36.927542 25118 webserver.cc:460] Webserver started at http://127.9.90.66:43777/ using document root <none> and password file <none>
I20260619 10:28:36.927764 25118 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:36.927816 25118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:36.927955 25118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:36.928962 25118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "307ad1c5818f47aaaafb7531c14cef84"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "479ad63b23533329a77613ddd9db9d41"
server_key_iv: "dc130000a8029c95edb506264a2c3026"
server_key_version: "encryptionkey@0"
I20260619 10:28:36.929294 25118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "307ad1c5818f47aaaafb7531c14cef84"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "479ad63b23533329a77613ddd9db9d41"
server_key_iv: "dc130000a8029c95edb506264a2c3026"
server_key_version: "encryptionkey@0"
I20260619 10:28:36.930675 25118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:28:36.931402 25133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:36.931566 25118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:28:36.931644 25118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "307ad1c5818f47aaaafb7531c14cef84"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "479ad63b23533329a77613ddd9db9d41"
server_key_iv: "dc130000a8029c95edb506264a2c3026"
server_key_version: "encryptionkey@0"
I20260619 10:28:36.931706 25118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:36.944278 25118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:36.946322 25118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:36.946456 25118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:36.946655 25118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:36.946946 25118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:36.946977 25118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:36.947005 25118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:36.947048 25118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:36.952886 25118 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:41175
I20260619 10:28:36.952941 25246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:41175 every 8 connection(s)
I20260619 10:28:36.953239 25118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:28:36 dist-test-slave-hw7q krb5kdc[24877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864916, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:36.961623 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:36.953877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:58453 (local address 127.9.90.126:36721)
0619 10:28:36.953938 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:36.953940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:36.954146 (+ 206us) server_negotiation.cc:407] Connection header received
0619 10:28:36.954643 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:36.954644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:36.954657 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:36.954705 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:36.955620 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.956126 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.957090 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.957290 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.957346 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:36.959385 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:36.959387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:36.959388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:36.959395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:36.960509 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:36.960868 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:36.960869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:36.960870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:36.960926 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:36.961118 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:36.961119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:36.961119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:36.961214 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:36.961266 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:36.961491 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 10:28:36.961558 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 10:28:36.961759 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:36.953873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:58453)
0619 10:28:36.954032 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:36.954067 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:36.954550 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:36.954788 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:36.954792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:36.955335 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:36.955525 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:36.955530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.956208 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.956209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:36.957006 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:36.957008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:36.957357 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:36.957358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:36.957478 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:36.957960 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:36.957968 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:36.959296 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:36.960582 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:36.960588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:36.960590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:36.960790 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:36.960987 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:36.960988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:36.960988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:36.961075 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:36.961258 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:36.961261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:36.961371 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 10:28:36.961469 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 10:28:36.961585 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260619 10:28:36.962165 25247 heartbeater.cc:347] Connected to a master server at 127.9.90.126:36721
I20260619 10:28:36.962253 25247 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:36.962385 25247 heartbeater.cc:511] Master 127.9.90.126:36721 requested a full tablet report, sending...
I20260619 10:28:36.962704 24922 ts_manager.cc:194] Registered new tserver with Master: 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175)
I20260619 10:28:36.962864 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 25118
I20260619 10:28:36.962956 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:28:36.963117 9577 external_mini_cluster.cc:1428] Setting key 6db0fc11097919038d5c39f7f3f1b76b
I20260619 10:28:36.963128 24922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:58453
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:28:36.984443 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36721
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:37.064538 25252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:37.064744 25252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:37.064767 25252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:37.065877 25252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:37.065912 25252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:37.065935 25252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:28:37.067291 25252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:37.067471 25252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:37.067773 25252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:37.072625 25257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:37.072664 25258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:37.072841 25252 server_base.cc:1034] running on GCE node
W20260619 10:28:37.072680 25260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:37.073071 25252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:37.073297 25252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:37.074471 25252 hybrid_clock.cc:648] HybridClock initialized: now 1781864917074451 us; error 33 us; skew 500 ppm
Jun 19 10:28:37 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864917, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:37.076978 25252 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:28:37.077402 25252 webserver.cc:460] Webserver started at http://127.9.90.67:40059/ using document root <none> and password file <none>
I20260619 10:28:37.077610 25252 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:37.077656 25252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:37.077770 25252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:28:37.078806 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "ecebd4ffb75d456fb997e8cea6beb41e"
format_stamp: "Formatted at 2026-06-19 10:28:37 on dist-test-slave-hw7q"
server_key: "78ed14e89e4747ffc6b3d2aabea03943"
server_key_iv: "2cce6635830ea1dd2b567d6549d564e2"
server_key_version: "encryptionkey@0"
I20260619 10:28:37.079137 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "ecebd4ffb75d456fb997e8cea6beb41e"
format_stamp: "Formatted at 2026-06-19 10:28:37 on dist-test-slave-hw7q"
server_key: "78ed14e89e4747ffc6b3d2aabea03943"
server_key_iv: "2cce6635830ea1dd2b567d6549d564e2"
server_key_version: "encryptionkey@0"
I20260619 10:28:37.080336 25252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:28:37.081001 25267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:37.081153 25252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:28:37.081233 25252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "ecebd4ffb75d456fb997e8cea6beb41e"
format_stamp: "Formatted at 2026-06-19 10:28:37 on dist-test-slave-hw7q"
server_key: "78ed14e89e4747ffc6b3d2aabea03943"
server_key_iv: "2cce6635830ea1dd2b567d6549d564e2"
server_key_version: "encryptionkey@0"
I20260619 10:28:37.081292 25252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:37.095709 25252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:37.097462 25252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:37.097577 25252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:37.097757 25252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:37.098067 25252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:37.098100 25252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:37.098127 25252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:37.098155 25252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:37.103636 25252 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:42495
I20260619 10:28:37.103689 25380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:42495 every 8 connection(s)
I20260619 10:28:37.103996 25252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:28:37.108965 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 25252
I20260619 10:28:37.109043 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:28:37.109203 9577 external_mini_cluster.cc:1428] Setting key 52c73ec2b46d6dd5ec99f880948a1369
Jun 19 10:28:37 dist-test-slave-hw7q krb5kdc[24877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864917, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:37.112453 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.104645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:52803 (local address 127.9.90.126:36721)
0619 10:28:37.104689 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:37.104690 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:37.104887 (+ 197us) server_negotiation.cc:407] Connection header received
0619 10:28:37.105328 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:37.105329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:37.105339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:37.105384 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:37.106332 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.106824 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.107847 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.108055 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.108109 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.110166 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:37.110167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:37.110167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:37.110174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:37.111319 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:37.111693 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:37.111693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:37.111694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:37.111743 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:37.111959 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:37.111960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:37.111960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:37.112057 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:37.112103 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:37.112306 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 10:28:37.112370 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:28:37.112633 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.104627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:52803)
0619 10:28:37.104780 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:37.104811 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:37.105259 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:37.105445 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:37.105448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:37.106021 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:37.106204 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.106209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.106929 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.106930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.107750 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.107752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.108125 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.108126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.108226 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.108626 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:37.108633 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:37.110081 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:37.111402 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:37.111407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:37.111409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:37.111625 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:37.111806 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:37.111807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:37.111807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:37.111911 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:37.112103 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:37.112105 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:37.112217 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 10:28:37.112292 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 10:28:37.112417 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
I20260619 10:28:37.113058 25381 heartbeater.cc:347] Connected to a master server at 127.9.90.126:36721
I20260619 10:28:37.113139 25381 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:37.113293 25381 heartbeater.cc:511] Master 127.9.90.126:36721 requested a full tablet report, sending...
I20260619 10:28:37.113669 24922 ts_manager.cc:194] Registered new tserver with Master: ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495)
I20260619 10:28:37.114077 24922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:52803
I20260619 10:28:37.120954 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:28:37.127702 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.122238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.9.90.126:36721)
0619 10:28:37.122287 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:37.122290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:37.122404 (+ 114us) server_negotiation.cc:407] Connection header received
0619 10:28:37.122458 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:37.122459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:37.122469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:37.122515 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:37.123088 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.123574 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.124256 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.124434 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.124481 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.125510 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:37.125511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:37.125511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:37.125517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:37.126826 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:37.127126 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:37.127126 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:37.127127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:37.127169 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:37.127292 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:37.127293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:37.127293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:37.127421 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:37.127490 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:37.127565 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 10:28:37.127617 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:28:37.129004 24922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:28:37.130576 24922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:28:37.137056 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.132700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.9.90.67:42495)
0619 10:28:37.133101 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:37.133106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:37.133117 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:28:37.133161 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:37.133163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:37.133180 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:37.133308 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:37.133558 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.134446 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.135976 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.136700 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.136781 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.136840 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:37.136967 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 10:28:37.136998 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":93,"threads_started":1}
I20260619 10:28:37.137090 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.132580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:42495 (local address 127.0.0.1:60600)
0619 10:28:37.132732 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:37.132781 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:37.132839 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:37.133402 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:37.133409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:37.133411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:37.133476 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.133478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.134533 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.134534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.135894 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.135897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.136783 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.136784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.136862 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.136877 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 10:28:37.136955 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 10:28:37.137009 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":30,"thread_start_us":51,"threads_started":1}
I20260619 10:28:37.137501 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.132611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:41175 (local address 127.0.0.1:44806)
0619 10:28:37.132860 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:37.132868 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:37.132903 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:37.133400 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:37.133403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:37.133408 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:37.133471 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.133473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.134793 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.134794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.135896 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.135897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.137276 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.137277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.137338 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.137341 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:28:37.137393 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 10:28:37.137431 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":12,"thread_start_us":76,"threads_started":1}
I20260619 10:28:37.137748 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.132903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.9.90.66:41175)
0619 10:28:37.133112 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:37.133114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:37.133122 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:28:37.133159 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:37.133161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:37.133176 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:37.133320 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:37.133575 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.134693 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.135976 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.136698 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.136791 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.136840 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:37.137614 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 10:28:37.137656 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260619 10:28:37.137815 25315 tablet_service.cc:1467] Processing CreateTablet for tablet 10aad68161ed456da2da451d8365c786 (DEFAULT_TABLE table=test-table [id=7db8cc59d6ea4c66a7301444ead49549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:37.138052 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.132766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:34201 (local address 127.0.0.1:46410)
0619 10:28:37.133070 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:37.133082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:37.133152 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:37.133403 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:37.133403 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:37.133405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:37.133471 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.133473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.136000 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.136001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.136703 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.136705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.137698 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.137699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.137779 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.137805 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 10:28:37.137948 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 10:28:37.137990 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":53,"threads_started":1}
I20260619 10:28:37.138188 25315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aad68161ed456da2da451d8365c786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:37.138265 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.132888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.9.90.65:34201)
0619 10:28:37.133046 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:37.133049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:37.133154 (+ 105us) server_negotiation.cc:407] Connection header received
0619 10:28:37.133230 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:37.133232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:37.133244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:37.133318 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:37.133547 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.134692 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.136803 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.137508 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.137581 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.137640 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:37.138146 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 10:28:37.138191 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260619 10:28:37.138523 25181 tablet_service.cc:1467] Processing CreateTablet for tablet 10aad68161ed456da2da451d8365c786 (DEFAULT_TABLE table=test-table [id=7db8cc59d6ea4c66a7301444ead49549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:37.138787 25181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aad68161ed456da2da451d8365c786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:37.138907 25047 tablet_service.cc:1467] Processing CreateTablet for tablet 10aad68161ed456da2da451d8365c786 (DEFAULT_TABLE table=test-table [id=7db8cc59d6ea4c66a7301444ead49549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:37.139184 25047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aad68161ed456da2da451d8365c786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:37.141089 25395 tablet_bootstrap.cc:492] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: Bootstrap starting.
I20260619 10:28:37.141772 25395 tablet_bootstrap.cc:654] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:37.141819 25396 tablet_bootstrap.cc:492] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: Bootstrap starting.
I20260619 10:28:37.142151 25397 tablet_bootstrap.cc:492] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Bootstrap starting.
I20260619 10:28:37.142467 25395 log.cc:824] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:37.142552 25396 tablet_bootstrap.cc:654] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:37.142688 25397 tablet_bootstrap.cc:654] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:37.143105 25396 log.cc:824] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:37.143234 25397 log.cc:824] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:37.143250 25395 tablet_bootstrap.cc:492] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: No bootstrap required, opened a new log
I20260619 10:28:37.143321 25395 ts_tablet_manager.cc:1397] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:28:37.143853 25396 tablet_bootstrap.cc:492] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: No bootstrap required, opened a new log
I20260619 10:28:37.143914 25396 ts_tablet_manager.cc:1397] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:28:37.144019 25397 tablet_bootstrap.cc:492] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: No bootstrap required, opened a new log
I20260619 10:28:37.144078 25397 ts_tablet_manager.cc:1397] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:28:37.145221 25396 raft_consensus.cc:348] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:37.145345 25396 raft_consensus.cc:374] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:37.145364 25396 raft_consensus.cc:729] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecebd4ffb75d456fb997e8cea6beb41e, State: Initialized, Role: FOLLOWER
I20260619 10:28:37.145454 25396 consensus_queue.cc:260] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:37.145349 25395 raft_consensus.cc:348] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:37.145526 25395 raft_consensus.cc:374] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:37.145560 25395 raft_consensus.cc:729] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307ad1c5818f47aaaafb7531c14cef84, State: Initialized, Role: FOLLOWER
I20260619 10:28:37.145687 25396 ts_tablet_manager.cc:1428] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:28:37.145704 25381 heartbeater.cc:503] Master 127.9.90.126:36721 was elected leader, sending a full tablet report...
I20260619 10:28:37.145666 25395 consensus_queue.cc:260] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:37.145965 25395 ts_tablet_manager.cc:1428] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:28:37.146091 25247 heartbeater.cc:503] Master 127.9.90.126:36721 was elected leader, sending a full tablet report...
I20260619 10:28:37.146049 25397 raft_consensus.cc:348] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:37.146200 25397 raft_consensus.cc:374] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:37.146248 25397 raft_consensus.cc:729] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad281fd2bdae498180ad14ee0f9a8a70, State: Initialized, Role: FOLLOWER
I20260619 10:28:37.146373 25397 consensus_queue.cc:260] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:37.146638 25397 ts_tablet_manager.cc:1428] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:28:37.146752 25113 heartbeater.cc:503] Master 127.9.90.126:36721 was elected leader, sending a full tablet report...
W20260619 10:28:37.203769 25248 tablet.cc:2369] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:28:37.311246 25114 tablet.cc:2369] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 10:28:37.354626 25382 tablet.cc:2369] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:28:37.570641 25401 raft_consensus.cc:482] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:37.570793 25401 raft_consensus.cc:504] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:37.571161 25401 leader_election.cc:288] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad281fd2bdae498180ad14ee0f9a8a70 (127.9.90.65:34201), 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175)
I20260619 10:28:37.573918 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.571407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:52195 (local address 127.9.90.65:34201)
0619 10:28:37.571455 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:37.571457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:37.571657 (+ 200us) server_negotiation.cc:407] Connection header received
0619 10:28:37.571710 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:37.571711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:37.571724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:37.571778 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:37.571910 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.572361 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.573247 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.573592 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.573649 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.573658 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:37.573822 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 10:28:37.573861 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:28:37.573915 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.571441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:56055 (local address 127.9.90.66:41175)
0619 10:28:37.571490 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:37.571492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:37.571499 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:28:37.571538 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:37.571539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:37.571551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:37.571608 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:37.571790 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.572335 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.573227 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.573578 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.573642 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.573652 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:37.573815 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 10:28:37.573852 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:28:37.573913 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.571350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:41175 (local address 127.9.90.67:56055)
0619 10:28:37.571416 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:37.571429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:37.571477 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:37.571632 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:37.571632 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:37.571634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:37.571726 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.571728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.572409 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.572410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.573144 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.573146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.573651 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.573652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.573727 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.573730 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:28:37.573771 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 10:28:37.573829 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260619 10:28:37.573928 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.571351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:34201 (local address 127.9.90.67:52195)
0619 10:28:37.571592 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:37.571602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:37.571654 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:37.571785 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:37.571786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:37.571788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:37.571850 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.571851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.572439 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.572440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.573160 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:37.573162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.573663 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.573664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:37.573727 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.573731 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:28:37.573805 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:28:37.573842 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":60,"thread_start_us":101,"threads_started":1}
I20260619 10:28:37.574260 25201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aad68161ed456da2da451d8365c786" candidate_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307ad1c5818f47aaaafb7531c14cef84" is_pre_election: true
I20260619 10:28:37.574432 25201 raft_consensus.cc:2454] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecebd4ffb75d456fb997e8cea6beb41e in term 0.
I20260619 10:28:37.574478 25067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aad68161ed456da2da451d8365c786" candidate_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" is_pre_election: true
I20260619 10:28:37.574620 25067 raft_consensus.cc:2454] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecebd4ffb75d456fb997e8cea6beb41e in term 0.
I20260619 10:28:37.574643 25270 leader_election.cc:302] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307ad1c5818f47aaaafb7531c14cef84, ecebd4ffb75d456fb997e8cea6beb41e; no voters:
I20260619 10:28:37.574764 25401 raft_consensus.cc:2781] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:28:37.574819 25401 raft_consensus.cc:482] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:28:37.574836 25401 raft_consensus.cc:3037] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:37.575519 25401 raft_consensus.cc:504] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:37.575635 25401 leader_election.cc:288] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [CANDIDATE]: Term 1 election: Requested vote from peers ad281fd2bdae498180ad14ee0f9a8a70 (127.9.90.65:34201), 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175)
I20260619 10:28:37.575827 25067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aad68161ed456da2da451d8365c786" candidate_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad281fd2bdae498180ad14ee0f9a8a70"
I20260619 10:28:37.575841 25201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aad68161ed456da2da451d8365c786" candidate_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307ad1c5818f47aaaafb7531c14cef84"
I20260619 10:28:37.575914 25201 raft_consensus.cc:3037] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:37.575914 25067 raft_consensus.cc:3037] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:37.576766 25201 raft_consensus.cc:2454] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecebd4ffb75d456fb997e8cea6beb41e in term 1.
I20260619 10:28:37.576783 25067 raft_consensus.cc:2454] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecebd4ffb75d456fb997e8cea6beb41e in term 1.
I20260619 10:28:37.576920 25270 leader_election.cc:302] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307ad1c5818f47aaaafb7531c14cef84, ecebd4ffb75d456fb997e8cea6beb41e; no voters:
I20260619 10:28:37.577023 25401 raft_consensus.cc:2781] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:28:37.577159 25401 raft_consensus.cc:686] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 1 LEADER]: Becoming Leader. State: Replica: ecebd4ffb75d456fb997e8cea6beb41e, State: Running, Role: LEADER
I20260619 10:28:37.577250 25401 consensus_queue.cc:237] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:37.578027 24921 catalog_manager.cc:5495] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67). New cstate: current_term: 1 leader_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } health_report { overall_health: UNKNOWN } } }
I20260619 10:28:37.688144 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.685653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.9.90.67:42495)
0619 10:28:37.685776 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:37.685778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:37.685809 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:28:37.685927 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:37.685928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:37.685951 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:37.686024 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:28:37.686147 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.686655 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.687417 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:37.687630 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:37.687682 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:37.687828 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:28:37.687939 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:28:37.688010 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:37.688030 (+ 20us) server_negotiation.cc:299] Negotiation successful
0619 10:28:37.688058 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260619 10:28:37.689867 25067 raft_consensus.cc:1261] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Refusing update from remote peer ecebd4ffb75d456fb997e8cea6beb41e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:37.689877 25201 raft_consensus.cc:1261] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [term 1 FOLLOWER]: Refusing update from remote peer ecebd4ffb75d456fb997e8cea6beb41e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:37.690209 25401 consensus_queue.cc:1035] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:37.690346 25405 consensus_queue.cc:1035] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:37.692720 25413 mvcc.cc:205] Tried to move back new op lower bound from 7298518702855196672 to 7298518702396874752. Current Snapshot: MvccSnapshot[applied={T|T < 7298518702855196672}]
I20260619 10:28:37.692931 25414 mvcc.cc:205] Tried to move back new op lower bound from 7298518702855196672 to 7298518702396874752. Current Snapshot: MvccSnapshot[applied={T|T < 7298518702855196672}]
I20260619 10:28:37.694962 24922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:37.695042 24922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:37.695698 24922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Sending DeleteTablet for 3 replicas of tablet 10aad68161ed456da2da451d8365c786
I20260619 10:28:37.695991 25315 tablet_service.cc:1514] Processing DeleteTablet for tablet 10aad68161ed456da2da451d8365c786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:37 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:60600
I20260619 10:28:37.696015 25047 tablet_service.cc:1514] Processing DeleteTablet for tablet 10aad68161ed456da2da451d8365c786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:37 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46410
I20260619 10:28:37.696046 25181 tablet_service.cc:1514] Processing DeleteTablet for tablet 10aad68161ed456da2da451d8365c786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:37 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:44806
I20260619 10:28:37.696218 25422 tablet_replica.cc:330] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: stopping tablet replica
I20260619 10:28:37.696307 25422 raft_consensus.cc:2229] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:28:37.696297 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 24984
I20260619 10:28:37.696431 25422 raft_consensus.cc:2258] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:37.696794 25422 ts_tablet_manager.cc:1905] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:37.697631 25424 tablet_replica.cc:330] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: stopping tablet replica
I20260619 10:28:37.697749 25424 raft_consensus.cc:2229] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:28:37.697903 25424 raft_consensus.cc:2258] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:37.698357 25422 ts_tablet_manager.cc:1918] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:37.698454 25422 log.cc:1196] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/10aad68161ed456da2da451d8365c786
I20260619 10:28:37.698688 25422 ts_tablet_manager.cc:1939] T 10aad68161ed456da2da451d8365c786 P ecebd4ffb75d456fb997e8cea6beb41e: Deleting consensus metadata
I20260619 10:28:37.699115 24907 catalog_manager.cc:4841] TS ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495): tablet 10aad68161ed456da2da451d8365c786 (table test-table [id=7db8cc59d6ea4c66a7301444ead49549]) successfully deleted
I20260619 10:28:37.699391 25424 ts_tablet_manager.cc:1905] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:37.700421 25424 ts_tablet_manager.cc:1918] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:37.700479 25424 log.cc:1196] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/10aad68161ed456da2da451d8365c786
I20260619 10:28:37.700671 25424 ts_tablet_manager.cc:1939] T 10aad68161ed456da2da451d8365c786 P 307ad1c5818f47aaaafb7531c14cef84: Deleting consensus metadata
I20260619 10:28:37.701020 24908 catalog_manager.cc:4841] TS 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175): tablet 10aad68161ed456da2da451d8365c786 (table test-table [id=7db8cc59d6ea4c66a7301444ead49549]) successfully deleted
W20260619 10:28:37.701399 24907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:34201 (error 108)
I20260619 10:28:37.701872 25426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:37.701703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:34201 (local address 127.0.0.1:46424)
0619 10:28:37.701781 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:37.701834 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:34201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":36,"threads_started":1}
I20260619 10:28:37.701931 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 25118
W20260619 10:28:37.702003 24907 catalog_manager.cc:4584] TS ad281fd2bdae498180ad14ee0f9a8a70 (127.9.90.65:34201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10aad68161ed456da2da451d8365c786: Network error: Client connection negotiation failed: client connection to 127.9.90.65:34201: connect: Connection refused (error 111)
I20260619 10:28:37.706950 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 25252
I20260619 10:28:37.710947 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 24890
I20260619 10:28:57.715196 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36721
--webserver_interface=127.9.90.126
--webserver_port=33515
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:36721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:57.795324 25427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:57.795535 25427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:57.795558 25427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:57.796711 25427 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 10:28:57.796756 25427 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 10:28:57.796770 25427 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 10:28:57.796797 25427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:28:57.796825 25427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:57.796854 25427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:57.796881 25427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:28:57.796911 25427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:28:57.798439 25427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:36721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=33515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:57.798683 25427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:57.798890 25427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:57.807610 25434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:57.807694 25427 server_base.cc:1034] running on GCE node
W20260619 10:28:57.807641 25433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:57.807580 25436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:57.808418 25427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:57.808704 25427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:57.809846 25427 hybrid_clock.cc:648] HybridClock initialized: now 1781864937809813 us; error 46 us; skew 500 ppm
Jun 19 10:28:57 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:57.812306 25427 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:28:57.812743 25427 webserver.cc:460] Webserver started at http://127.9.90.126:33515/ using document root <none> and password file <none>
I20260619 10:28:57.812963 25427 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:57.813020 25427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:57.814459 25427 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:57.815141 25443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:57.815296 25427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:57.815408 25427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "b2d6d71cc0cc4fbdaee619f58138d65d"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "7e4b946570cfcb2e0c1e29403a9981c8"
server_key_iv: "f430dfed7ed006be2fef8dd18132cc68"
server_key_version: "encryptionkey@0"
I20260619 10:28:57.815786 25427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:57.839486 25427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:57.841219 25427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:57.841327 25427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:57.844995 25427 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:36721
I20260619 10:28:57.845045 25495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:36721 every 8 connection(s)
I20260619 10:28:57.845394 25427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:28:57.846833 25496 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:28:57.847409 25496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Bootstrap starting.
I20260619 10:28:57.850193 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 25427
I20260619 10:28:57.850438 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:34201
--local_ip_for_outbound_sockets=127.9.90.65
--tserver_master_addrs=127.9.90.126:36721
--webserver_port=46057
--webserver_interface=127.9.90.65
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:57.855235 25496 log.cc:824] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:57.856910 25496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:28:57.857136 25496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Bootstrap complete.
I20260619 10:28:57.859177 25496 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:28:57.859392 25496 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d6d71cc0cc4fbdaee619f58138d65d, State: Initialized, Role: FOLLOWER
I20260619 10:28:57.859535 25496 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:28:57.859603 25496 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:28:57.859661 25496 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:28:57.859709 25496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 1 FOLLOWER]: Advancing to term 2
I20260619 10:28:57.860615 25496 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:28:57.860718 25496 leader_election.cc:302] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d6d71cc0cc4fbdaee619f58138d65d; no voters:
I20260619 10:28:57.860877 25496 leader_election.cc:288] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 10:28:57.860915 25500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 10:28:57.861023 25500 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 2 LEADER]: Becoming Leader. State: Replica: b2d6d71cc0cc4fbdaee619f58138d65d, State: Running, Role: LEADER
I20260619 10:28:57.861124 25496 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:28:57.861182 25500 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:28:57.861519 25501 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } } }
I20260619 10:28:57.861528 25502 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d6d71cc0cc4fbdaee619f58138d65d. Latest consensus state: current_term: 2 leader_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } } }
I20260619 10:28:57.861609 25501 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:57.861639 25502 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: This master's current role is: LEADER
I20260619 10:28:57.862067 25507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:28:57.862846 25507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:28:57.863011 25507 catalog_manager.cc:1259] Loaded cluster ID: 99e0305724d2431bad8a7172d27c4956
I20260619 10:28:57.863219 25507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:28:57.864058 25507 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:28:57.864226 25507 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Loaded TSK: 0
I20260619 10:28:57.869732 25507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Generated new TSK 1
I20260619 10:28:57.869827 25507 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:28:57.949602 25498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:57.949867 25498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:57.949895 25498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:57.951061 25498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:57.951107 25498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:57.951143 25498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:28:57.952590 25498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:34201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=46057
--tserver_master_addrs=127.9.90.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:57.952805 25498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:57.953030 25498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:57.957993 25524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:57.958045 25498 server_base.cc:1034] running on GCE node
W20260619 10:28:57.958143 25526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:57.958052 25523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:57.958458 25498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:57.958699 25498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:57.959898 25498 hybrid_clock.cc:648] HybridClock initialized: now 1781864937959881 us; error 33 us; skew 500 ppm
Jun 19 10:28:57 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:57.962661 25498 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:28:57.963084 25498 webserver.cc:460] Webserver started at http://127.9.90.65:46057/ using document root <none> and password file <none>
I20260619 10:28:57.963318 25498 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:57.963375 25498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:57.964717 25498 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:57.965353 25533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:57.965508 25498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:28:57.965584 25498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "ad281fd2bdae498180ad14ee0f9a8a70"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "c00d52d4441aced37465fca6ecd99b7b"
server_key_iv: "e9cd184224bd228b302c4052c15c97b0"
server_key_version: "encryptionkey@0"
I20260619 10:28:57.965972 25498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:57.984097 25498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:57.986357 25498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:57.986479 25498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:57.986707 25498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:57.987133 25540 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 10:28:57.988978 25498 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 10:28:57.989027 25498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 10:28:57.989060 25498 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 10:28:57.989661 25498 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 10:28:57.989693 25498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 10:28:57.989820 25540 tablet_bootstrap.cc:492] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Bootstrap starting.
I20260619 10:28:57.995342 25498 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:34201
I20260619 10:28:57.995594 25647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:34201 every 8 connection(s)
I20260619 10:28:57.995801 25498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:28:58.000015 25540 log.cc:824] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:58.002457 25540 tablet_bootstrap.cc:492] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:28:58.002781 25540 tablet_bootstrap.cc:492] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Bootstrap complete.
Jun 19 10:28:57 dist-test-slave-hw7q krb5kdc[24877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:58.003561 25540 ts_tablet_manager.cc:1397] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260619 10:28:58.004786 25540 raft_consensus.cc:348] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:58.005120 25540 raft_consensus.cc:729] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad281fd2bdae498180ad14ee0f9a8a70, State: Initialized, Role: FOLLOWER
I20260619 10:28:58.005262 25540 consensus_queue.cc:260] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } } peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } }
I20260619 10:28:58.005367 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 25498
I20260619 10:28:58.005538 25540 ts_tablet_manager.cc:1428] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:28:58.005658 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:41175
--local_ip_for_outbound_sockets=127.9.90.66
--tserver_master_addrs=127.9.90.126:36721
--webserver_port=43777
--webserver_interface=127.9.90.66
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:28:58.009189 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:57.996578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:44703)
0619 10:28:57.996923 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:57.996954 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:57.997389 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:57.997804 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:57.997808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:57.998060 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:57.998237 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:57.998241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:57.999540 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:57.999542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.000452 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.000455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.001274 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.001275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.001406 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.002025 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:58.002034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:58.003817 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:58.007495 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:58.007502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:58.007504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:58.007732 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:58.007972 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:58.007973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:58.007973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:58.008080 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:58.008572 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:58.008578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:58.008725 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 10:28:58.008815 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 10:28:58.008986 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":86,"threads_started":1}
I20260619 10:28:58.009218 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:57.996664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:44703 (local address 127.9.90.126:36721)
0619 10:28:57.996853 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:57.996860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:57.997052 (+ 192us) server_negotiation.cc:407] Connection header received
0619 10:28:57.997593 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:57.997598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:57.997641 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:57.997750 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:57.998341 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:57.999283 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.000552 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.001160 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.001271 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.003938 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:58.003942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:58.003942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:58.003954 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:58.007406 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.007814 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.007817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.007817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.007908 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.008160 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.008161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.008161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.008480 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:58.008583 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.008885 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.009017 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260619 10:28:58.009954 25648 heartbeater.cc:347] Connected to a master server at 127.9.90.126:36721
I20260619 10:28:58.010040 25648 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:58.010196 25648 heartbeater.cc:511] Master 127.9.90.126:36721 requested a full tablet report, sending...
I20260619 10:28:58.010818 25460 ts_manager.cc:194] Registered new tserver with Master: ad281fd2bdae498180ad14ee0f9a8a70 (127.9.90.65:34201)
I20260619 10:28:58.011780 25460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:44703
I20260619 10:28:58.012560 25648 heartbeater.cc:503] Master 127.9.90.126:36721 was elected leader, sending a full tablet report...
I20260619 10:28:58.017377 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.011342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:34201 (local address 127.0.0.1:49108)
0619 10:28:58.013903 (+ 2561us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:58.013944 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:58.014012 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:58.014304 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:58.014307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:58.014309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:58.014357 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.014359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.015521 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.015522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.016300 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.016303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.017162 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.017163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.017220 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.017229 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 10:28:58.017289 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 10:28:58.017325 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2527,"thread_start_us":79,"threads_started":1}
I20260619 10:28:58.017424 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.012044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.9.90.65:34201)
0619 10:28:58.014144 (+ 2100us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.014149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.014158 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:28:58.014189 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.014191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.014207 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.014285 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:58.014440 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.015448 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.016385 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.017082 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.017152 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.017201 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.017316 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.017358 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2064,"thread_start_us":91,"threads_started":1}
I20260619 10:28:58.017746 25578 tablet_service.cc:1514] Processing DeleteTablet for tablet 10aad68161ed456da2da451d8365c786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:37 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:49108
I20260619 10:28:58.018270 25657 tablet_replica.cc:330] stopping tablet replica
I20260619 10:28:58.018379 25657 raft_consensus.cc:2229] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:28:58.018442 25657 raft_consensus.cc:2258] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:58.018730 25657 ts_tablet_manager.cc:1905] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:58.020792 25657 ts_tablet_manager.cc:1918] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:58.020875 25657 log.cc:1196] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal/wals/10aad68161ed456da2da451d8365c786
I20260619 10:28:58.021097 25657 ts_tablet_manager.cc:1939] T 10aad68161ed456da2da451d8365c786 P ad281fd2bdae498180ad14ee0f9a8a70: Deleting consensus metadata
I20260619 10:28:58.021704 25445 catalog_manager.cc:4841] TS ad281fd2bdae498180ad14ee0f9a8a70 (127.9.90.65:34201): tablet 10aad68161ed456da2da451d8365c786 (table test-table [id=7db8cc59d6ea4c66a7301444ead49549]) successfully deleted
W20260619 10:28:58.086581 25654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:58.086769 25654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:58.086791 25654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:58.087875 25654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:58.087909 25654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:58.087934 25654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:28:58.089329 25654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:41175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=43777
--tserver_master_addrs=127.9.90.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:58.089495 25654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:58.089660 25654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:58.094497 25662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:58.094477 25665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:58.094507 25654 server_base.cc:1034] running on GCE node
W20260619 10:28:58.094493 25663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:58.094856 25654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:58.095044 25654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:58.096199 25654 hybrid_clock.cc:648] HybridClock initialized: now 1781864938096187 us; error 28 us; skew 500 ppm
Jun 19 10:28:58 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864938, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:58.098476 25654 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:28:58.098886 25654 webserver.cc:460] Webserver started at http://127.9.90.66:43777/ using document root <none> and password file <none>
I20260619 10:28:58.099067 25654 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:58.099123 25654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:58.100376 25654 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:28:58.100914 25672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:58.101066 25654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:58.101141 25654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "307ad1c5818f47aaaafb7531c14cef84"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "479ad63b23533329a77613ddd9db9d41"
server_key_iv: "dc130000a8029c95edb506264a2c3026"
server_key_version: "encryptionkey@0"
I20260619 10:28:58.101451 25654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:58.112357 25654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:58.114210 25654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:58.114320 25654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:58.114490 25654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:58.114719 25654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:58.114758 25654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:58.114777 25654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:58.114794 25654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:58.120111 25654 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:41175
I20260619 10:28:58.120169 25785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:41175 every 8 connection(s)
I20260619 10:28:58.120460 25654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:28:58 dist-test-slave-hw7q krb5kdc[24877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864938, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:58.128337 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.121123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:45013 (local address 127.9.90.126:36721)
0619 10:28:58.121166 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.121168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.121392 (+ 224us) server_negotiation.cc:407] Connection header received
0619 10:28:58.121884 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.121885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.121900 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.121947 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:58.122771 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.123261 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.124167 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.124380 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.124432 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.126300 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:58.126302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:58.126302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:58.126308 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:58.127357 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.127670 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.127671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.127671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.127714 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.127905 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.127906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.127906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.127996 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:58.128038 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.128216 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.128275 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:28:58.128487 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.121129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:45013)
0619 10:28:58.121281 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:58.121314 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:58.121750 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:58.122010 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:58.122013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:58.122529 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:58.122703 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.122707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.123326 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.123326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.124094 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.124096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.124439 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.124440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.124531 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.124957 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:58.124968 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:58.126224 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:58.127419 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:58.127425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:58.127427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:58.127616 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:58.127774 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:58.127775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:58.127775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:58.127864 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:58.128036 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:58.128038 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:58.128136 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 10:28:58.128206 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 10:28:58.128310 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260619 10:28:58.128917 25786 heartbeater.cc:347] Connected to a master server at 127.9.90.126:36721
I20260619 10:28:58.128998 25786 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:58.129130 25786 heartbeater.cc:511] Master 127.9.90.126:36721 requested a full tablet report, sending...
I20260619 10:28:58.129460 25460 ts_manager.cc:194] Registered new tserver with Master: 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175)
I20260619 10:28:58.129864 25460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:45013
I20260619 10:28:58.130553 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 25654
I20260619 10:28:58.130785 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:42495
--local_ip_for_outbound_sockets=127.9.90.67
--tserver_master_addrs=127.9.90.126:36721
--webserver_port=40059
--webserver_interface=127.9.90.67
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:28:58.208362 25789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:28:58.208566 25789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:28:58.208598 25789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:28:58.209724 25789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:28:58.209767 25789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:28:58.209820 25789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:28:58.211230 25789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:42495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=40059
--tserver_master_addrs=127.9.90.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:28:58.211470 25789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:28:58.211668 25789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:28:58.216297 25797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:58.216364 25795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:28:58.216372 25794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:28:58.216502 25789 server_base.cc:1034] running on GCE node
I20260619 10:28:58.216696 25789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:28:58.216913 25789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:28:58.218040 25789 hybrid_clock.cc:648] HybridClock initialized: now 1781864938218034 us; error 30 us; skew 500 ppm
Jun 19 10:28:58 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864938, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:28:58.220428 25789 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:28:58.220840 25789 webserver.cc:460] Webserver started at http://127.9.90.67:40059/ using document root <none> and password file <none>
I20260619 10:28:58.221110 25789 fs_manager.cc:359] Metadata directory not provided
I20260619 10:28:58.221167 25789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:28:58.222484 25789 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:28:58.222997 25804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:58.223127 25789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:58.223198 25789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "ecebd4ffb75d456fb997e8cea6beb41e"
format_stamp: "Formatted at 2026-06-19 10:28:37 on dist-test-slave-hw7q"
server_key: "78ed14e89e4747ffc6b3d2aabea03943"
server_key_iv: "2cce6635830ea1dd2b567d6549d564e2"
server_key_version: "encryptionkey@0"
I20260619 10:28:58.223497 25789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:28:58.231038 25789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:28:58.232601 25789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:28:58.232712 25789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:28:58.232895 25789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:28:58.233147 25789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:28:58.233177 25789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:58.233198 25789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:28:58.233220 25789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:58.238870 25789 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:42495
I20260619 10:28:58.238909 25917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:42495 every 8 connection(s)
I20260619 10:28:58.239214 25789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
Jun 19 10:28:58 dist-test-slave-hw7q krb5kdc[24877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864938, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:28:58.244868 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 25789
I20260619 10:28:58.247898 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.239874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:47529 (local address 127.9.90.126:36721)
0619 10:28:58.239934 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.239936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.240106 (+ 170us) server_negotiation.cc:407] Connection header received
0619 10:28:58.240556 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.240557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.240567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.240617 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:58.241489 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.242013 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.242924 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.243106 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.243153 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.245226 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:58.245228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:58.245228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:58.245235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:58.246674 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.247056 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.247056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.247057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.247107 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.247348 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.247348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.247348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.247438 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:58.247480 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.247772 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.247838 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:28:58.248118 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.239855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:47529)
0619 10:28:58.239992 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:58.240031 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:58.240479 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:58.240675 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:58.240679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:58.241234 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:28:58.241423 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.241427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.242084 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.242084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.242856 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.242858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.243193 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.243194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.243305 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.243749 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:28:58.243757 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:28:58.245136 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:28:58.246754 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:58.246759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:58.246761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:58.246987 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:58.247158 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:28:58.247159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:28:58.247159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:28:58.247289 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:28:58.247489 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:28:58.247493 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:28:58.247642 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 10:28:58.247749 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 10:28:58.247909 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260619 10:28:58.248584 25918 heartbeater.cc:347] Connected to a master server at 127.9.90.126:36721
I20260619 10:28:58.248675 25918 heartbeater.cc:464] Registering TS with master...
I20260619 10:28:58.248857 25918 heartbeater.cc:511] Master 127.9.90.126:36721 requested a full tablet report, sending...
I20260619 10:28:58.249250 25460 ts_manager.cc:194] Registered new tserver with Master: ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495)
I20260619 10:28:58.249641 25460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:47529
I20260619 10:28:58.251904 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.245448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.9.90.126:36721)
0619 10:28:58.245551 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.245553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.245559 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:28:58.245577 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.245578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.245588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.245658 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:58.246324 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.246969 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.247996 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.248278 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.248344 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.249524 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:58.249525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:58.249526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:58.249532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:58.251009 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.251297 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.251298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.251298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.251350 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.251477 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.251477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.251477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.251566 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:58.251693 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.251722 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.251790 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
I20260619 10:28:58.252631 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:28:58.255317 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.252986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.9.90.126:36721)
0619 10:28:58.253039 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.253041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.253094 (+ 53us) server_negotiation.cc:407] Connection header received
0619 10:28:58.253167 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.253168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.253209 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.253267 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:28:58.253374 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.253870 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.254566 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.254766 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.254817 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.254956 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:28:58.254999 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 10:28:58.255255 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58108: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 10:28:58.255368 25922 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58108: token expired
W20260619 10:28:58.255700 25384 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.90.126:36721)
I20260619 10:28:58.262382 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.255963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.9.90.126:36721)
0619 10:28:58.256027 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.256028 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.256035 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:28:58.256052 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.256053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.256060 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.256105 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:28:58.256529 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.256963 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.257820 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.258120 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.258202 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.259419 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:28:58.259420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:28:58.259421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:28:58.259442 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:28:58.261137 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.261495 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.261496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.261496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.261559 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:28:58.261747 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:28:58.261748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:28:58.261749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:28:58.261999 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:28:58.262089 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.262211 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.262285 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 10:28:58.263466 25460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:28:58.264972 25460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:28:58.267364 25578 tablet_service.cc:1467] Processing CreateTablet for tablet 6aae1067db794d438fbe36d7164459e2 (DEFAULT_TABLE table=test-table [id=85cf1b59c16044e891b02e2b5ef2838b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:58.267576 25578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aae1067db794d438fbe36d7164459e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:58.269119 25540 tablet_bootstrap.cc:492] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Bootstrap starting.
I20260619 10:28:58.269451 25540 tablet_bootstrap.cc:654] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:58.270097 25540 tablet_bootstrap.cc:492] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: No bootstrap required, opened a new log
I20260619 10:28:58.270169 25540 ts_tablet_manager.cc:1397] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 10:28:58.270275 25540 raft_consensus.cc:348] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:28:58.270344 25540 raft_consensus.cc:374] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:58.270386 25540 raft_consensus.cc:729] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad281fd2bdae498180ad14ee0f9a8a70, State: Initialized, Role: FOLLOWER
I20260619 10:28:58.270434 25540 consensus_queue.cc:260] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:28:58.270502 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.266823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:42495 (local address 127.0.0.1:53626)
0619 10:28:58.267034 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:58.267042 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:58.267069 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:58.267417 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:58.267418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:58.267419 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:58.267443 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.267444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.268626 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.268627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.269600 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.269602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.270266 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.270267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.270327 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.270331 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:28:58.270392 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:28:58.270435 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
I20260619 10:28:58.270519 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.267061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.9.90.67:42495)
0619 10:28:58.267232 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.267237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.267247 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:28:58.267286 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.267288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.267304 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.267422 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:58.267548 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.268544 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.269665 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.270195 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.270260 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.270316 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.270406 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.270441 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260619 10:28:58.270526 25540 ts_tablet_manager.cc:1428] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 10:28:58.270691 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.266794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:41175 (local address 127.0.0.1:38686)
0619 10:28:58.266981 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:58.266991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:58.267040 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:58.267298 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:58.267299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:58.267301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:58.267341 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.267342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.268324 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.268325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.269343 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.269346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.270435 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.270437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.270501 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.270505 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:28:58.270561 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 10:28:58.270602 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260619 10:28:58.270917 25925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.266868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.9.90.66:41175)
0619 10:28:58.267043 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.267048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.267061 (+ 13us) server_negotiation.cc:407] Connection header received
0619 10:28:58.267104 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.267106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.267124 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.267204 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:58.267408 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.268236 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.269481 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.270254 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.270333 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.270394 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.270790 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.270838 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260619 10:28:58.271284 25852 tablet_service.cc:1467] Processing CreateTablet for tablet 6aae1067db794d438fbe36d7164459e2 (DEFAULT_TABLE table=test-table [id=85cf1b59c16044e891b02e2b5ef2838b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:58.271612 25852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aae1067db794d438fbe36d7164459e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:58.271544 25720 tablet_service.cc:1467] Processing CreateTablet for tablet 6aae1067db794d438fbe36d7164459e2 (DEFAULT_TABLE table=test-table [id=85cf1b59c16044e891b02e2b5ef2838b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:28:58.271829 25720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aae1067db794d438fbe36d7164459e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:28:58.274925 25928 tablet_bootstrap.cc:492] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: Bootstrap starting.
I20260619 10:28:58.275522 25928 tablet_bootstrap.cc:654] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:58.275972 25928 log.cc:824] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:58.276291 25929 tablet_bootstrap.cc:492] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: Bootstrap starting.
I20260619 10:28:58.276681 25928 tablet_bootstrap.cc:492] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: No bootstrap required, opened a new log
I20260619 10:28:58.276741 25928 ts_tablet_manager.cc:1397] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:28:58.276897 25929 tablet_bootstrap.cc:654] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: Neither blocks nor log segments found. Creating new log.
I20260619 10:28:58.277379 25929 log.cc:824] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:28:58.278046 25929 tablet_bootstrap.cc:492] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: No bootstrap required, opened a new log
I20260619 10:28:58.278107 25929 ts_tablet_manager.cc:1397] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:28:58.278196 25928 raft_consensus.cc:348] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:28:58.278331 25928 raft_consensus.cc:374] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:58.278379 25928 raft_consensus.cc:729] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307ad1c5818f47aaaafb7531c14cef84, State: Initialized, Role: FOLLOWER
I20260619 10:28:58.278488 25928 consensus_queue.cc:260] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:28:58.278705 25928 ts_tablet_manager.cc:1428] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:28:58.278759 25786 heartbeater.cc:503] Master 127.9.90.126:36721 was elected leader, sending a full tablet report...
I20260619 10:28:58.279893 25929 raft_consensus.cc:348] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:28:58.280055 25929 raft_consensus.cc:374] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:28:58.280093 25929 raft_consensus.cc:729] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecebd4ffb75d456fb997e8cea6beb41e, State: Initialized, Role: FOLLOWER
I20260619 10:28:58.280213 25929 consensus_queue.cc:260] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:28:58.280444 25929 ts_tablet_manager.cc:1428] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 10:28:58.280488 25918 heartbeater.cc:503] Master 127.9.90.126:36721 was elected leader, sending a full tablet report...
I20260619 10:28:58.294771 25932 raft_consensus.cc:482] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:28:58.294869 25932 raft_consensus.cc:504] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:28:58.295231 25932 leader_election.cc:288] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495), ad281fd2bdae498180ad14ee0f9a8a70 (127.9.90.65:34201)
I20260619 10:28:58.298979 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.295574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:59053 (local address 127.9.90.65:34201)
0619 10:28:58.295625 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.295627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.295697 (+ 70us) server_negotiation.cc:407] Connection header received
0619 10:28:58.295749 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.295750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.295761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.295811 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:58.295936 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.296616 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.298058 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.298661 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.298726 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.298736 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.298874 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.298911 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:28:58.298979 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.295477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:46555 (local address 127.9.90.67:42495)
0619 10:28:58.295599 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.295601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.295609 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:28:58.295655 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.295656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.295668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.295726 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:28:58.295872 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.296594 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.298065 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.298643 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.298726 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.298736 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.298874 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.298911 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 10:28:58.298995 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.295397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:42495 (local address 127.9.90.66:46555)
0619 10:28:58.295456 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:58.295469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:58.295517 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:58.295736 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:58.295737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:58.295740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:58.295808 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.295809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.296685 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.296686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.297969 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.297971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.298726 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.298727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.298813 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.298815 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:28:58.298864 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:28:58.298907 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":85}
I20260619 10:28:58.299000 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.295507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:34201 (local address 127.9.90.66:59053)
0619 10:28:58.295637 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:58.295645 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:28:58.295693 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:28:58.295829 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:28:58.295829 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:28:58.295831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:28:58.295885 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.295886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.296685 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.296686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.297968 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:28:58.297971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.298733 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.298733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:28:58.298802 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.298807 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:28:58.298858 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:28:58.298907 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":34,"thread_start_us":78,"threads_started":1}
I20260619 10:28:58.299317 25872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aae1067db794d438fbe36d7164459e2" candidate_uuid: "307ad1c5818f47aaaafb7531c14cef84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" is_pre_election: true
I20260619 10:28:58.299466 25872 raft_consensus.cc:2454] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307ad1c5818f47aaaafb7531c14cef84 in term 0.
I20260619 10:28:58.299441 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aae1067db794d438fbe36d7164459e2" candidate_uuid: "307ad1c5818f47aaaafb7531c14cef84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" is_pre_election: true
I20260619 10:28:58.299563 25600 raft_consensus.cc:2454] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307ad1c5818f47aaaafb7531c14cef84 in term 0.
I20260619 10:28:58.299676 25674 leader_election.cc:302] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307ad1c5818f47aaaafb7531c14cef84, ecebd4ffb75d456fb997e8cea6beb41e; no voters:
I20260619 10:28:58.299795 25932 raft_consensus.cc:2781] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:28:58.299846 25932 raft_consensus.cc:482] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:28:58.299867 25932 raft_consensus.cc:3037] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:58.300467 25932 raft_consensus.cc:504] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:28:58.300566 25932 leader_election.cc:288] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [CANDIDATE]: Term 1 election: Requested vote from peers ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495), ad281fd2bdae498180ad14ee0f9a8a70 (127.9.90.65:34201)
I20260619 10:28:58.300711 25872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aae1067db794d438fbe36d7164459e2" candidate_uuid: "307ad1c5818f47aaaafb7531c14cef84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecebd4ffb75d456fb997e8cea6beb41e"
I20260619 10:28:58.300748 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aae1067db794d438fbe36d7164459e2" candidate_uuid: "307ad1c5818f47aaaafb7531c14cef84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad281fd2bdae498180ad14ee0f9a8a70"
I20260619 10:28:58.300793 25872 raft_consensus.cc:3037] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:58.300820 25600 raft_consensus.cc:3037] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:28:58.301437 25872 raft_consensus.cc:2454] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307ad1c5818f47aaaafb7531c14cef84 in term 1.
I20260619 10:28:58.301579 25674 leader_election.cc:302] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307ad1c5818f47aaaafb7531c14cef84, ecebd4ffb75d456fb997e8cea6beb41e; no voters:
I20260619 10:28:58.301651 25932 raft_consensus.cc:2781] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:28:58.301765 25932 raft_consensus.cc:686] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 1 LEADER]: Becoming Leader. State: Replica: 307ad1c5818f47aaaafb7531c14cef84, State: Running, Role: LEADER
I20260619 10:28:58.301868 25600 raft_consensus.cc:2454] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307ad1c5818f47aaaafb7531c14cef84 in term 1.
I20260619 10:28:58.301870 25932 consensus_queue.cc:237] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:28:58.302650 25460 catalog_manager.cc:5495] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66). New cstate: current_term: 1 leader_uuid: "307ad1c5818f47aaaafb7531c14cef84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } health_report { overall_health: UNKNOWN } } }
I20260619 10:28:58.307233 25925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.305185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.9.90.66:41175)
0619 10:28:58.305222 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:28:58.305224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:28:58.305233 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:28:58.305265 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:28:58.305266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:28:58.305284 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:28:58.305346 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:28:58.305447 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.305958 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.306652 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:28:58.306846 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:28:58.306900 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:28:58.307000 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:28:58.307084 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:28:58.307134 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 10:28:58.307173 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 10:28:58.307188 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:28:58.308729 25600 raft_consensus.cc:1261] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Refusing update from remote peer 307ad1c5818f47aaaafb7531c14cef84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:58.308748 25872 raft_consensus.cc:1261] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [term 1 FOLLOWER]: Refusing update from remote peer 307ad1c5818f47aaaafb7531c14cef84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:28:58.308993 25932 consensus_queue.cc:1035] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:58.309113 25935 consensus_queue.cc:1035] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:28:58.311510 25939 mvcc.cc:205] Tried to move back new op lower bound from 7298518787310505984 to 7298518787285106688. Current Snapshot: MvccSnapshot[applied={T|T < 7298518787310505984}]
I20260619 10:28:58.311576 25940 mvcc.cc:205] Tried to move back new op lower bound from 7298518787310505984 to 7298518787285106688. Current Snapshot: MvccSnapshot[applied={T|T < 7298518787310505984}]
I20260619 10:28:58.313514 25459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:58.313617 25459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:28:58.314301 25459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Sending DeleteTablet for 3 replicas of tablet 6aae1067db794d438fbe36d7164459e2
I20260619 10:28:58.314616 25578 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aae1067db794d438fbe36d7164459e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:58 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:49108
I20260619 10:28:58.314596 25852 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aae1067db794d438fbe36d7164459e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:58 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:53626
I20260619 10:28:58.314618 25720 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aae1067db794d438fbe36d7164459e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:28:58 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:38686
I20260619 10:28:58.314688 25657 tablet_replica.cc:330] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: stopping tablet replica
I20260619 10:28:58.314729 25657 raft_consensus.cc:2229] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:28:58.314778 25657 raft_consensus.cc:2258] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:58.314783 25948 tablet_replica.cc:330] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: stopping tablet replica
I20260619 10:28:58.314859 25948 raft_consensus.cc:2229] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:28:58.314945 25948 raft_consensus.cc:2258] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:58.314944 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 25498
I20260619 10:28:58.315806 25948 ts_tablet_manager.cc:1905] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:58.317277 25948 ts_tablet_manager.cc:1918] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:58.317344 25948 log.cc:1196] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/6aae1067db794d438fbe36d7164459e2
I20260619 10:28:58.317590 25948 ts_tablet_manager.cc:1939] T 6aae1067db794d438fbe36d7164459e2 P ecebd4ffb75d456fb997e8cea6beb41e: Deleting consensus metadata
I20260619 10:28:58.317902 25949 tablet_replica.cc:330] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: stopping tablet replica
I20260619 10:28:58.317997 25949 raft_consensus.cc:2229] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:28:58.318118 25949 raft_consensus.cc:2258] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:28:58.318121 25445 catalog_manager.cc:4841] TS ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495): tablet 6aae1067db794d438fbe36d7164459e2 (table test-table [id=85cf1b59c16044e891b02e2b5ef2838b]) successfully deleted
I20260619 10:28:58.318470 25949 ts_tablet_manager.cc:1905] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:28:58.320051 25949 ts_tablet_manager.cc:1918] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:28:58.320119 25949 log.cc:1196] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/6aae1067db794d438fbe36d7164459e2
I20260619 10:28:58.320376 25949 ts_tablet_manager.cc:1939] T 6aae1067db794d438fbe36d7164459e2 P 307ad1c5818f47aaaafb7531c14cef84: Deleting consensus metadata
I20260619 10:28:58.320741 25446 catalog_manager.cc:4841] TS 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175): tablet 6aae1067db794d438fbe36d7164459e2 (table test-table [id=85cf1b59c16044e891b02e2b5ef2838b]) successfully deleted
W20260619 10:28:58.321026 25445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:34201 (error 108)
I20260619 10:28:58.321424 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 25654
I20260619 10:28:58.321950 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:28:58.321362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:34201 (local address 127.0.0.1:49114)
0619 10:28:58.321875 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 10:28:58.321918 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:34201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492}
W20260619 10:28:58.322450 25445 catalog_manager.cc:4584] TS ad281fd2bdae498180ad14ee0f9a8a70 (127.9.90.65:34201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6aae1067db794d438fbe36d7164459e2: Network error: Client connection negotiation failed: client connection to 127.9.90.65:34201: connect: Connection refused (error 111)
I20260619 10:28:58.325666 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 25789
I20260619 10:28:58.329316 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 25427
I20260619 10:29:18.333853 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36721
--webserver_interface=127.9.90.126
--webserver_port=33515
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:36721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:18.412075 25951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:18.412282 25951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:18.412310 25951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:18.413421 25951 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 10:29:18.413458 25951 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 10:29:18.413475 25951 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 10:29:18.413489 25951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:18.413504 25951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:18.413522 25951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:18.413542 25951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:18.413564 25951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:18.415424 25951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:36721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=33515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:18.415670 25951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:18.415905 25951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:18.420933 25957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:18.420984 25959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:18.421126 25951 server_base.cc:1034] running on GCE node
W20260619 10:29:18.421232 25956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:18.421475 25951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:18.421753 25951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:18.422902 25951 hybrid_clock.cc:648] HybridClock initialized: now 1781864958422893 us; error 26 us; skew 500 ppm
Jun 19 10:29:18 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:18.425253 25951 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:29:18.425688 25951 webserver.cc:460] Webserver started at http://127.9.90.126:33515/ using document root <none> and password file <none>
I20260619 10:29:18.425925 25951 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:18.425976 25951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:18.427222 25951 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:29:18.427840 25966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.428007 25951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.428097 25951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "b2d6d71cc0cc4fbdaee619f58138d65d"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "7e4b946570cfcb2e0c1e29403a9981c8"
server_key_iv: "f430dfed7ed006be2fef8dd18132cc68"
server_key_version: "encryptionkey@0"
I20260619 10:29:18.428398 25951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:18.452689 25951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:18.454403 25951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:18.454530 25951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:18.458302 25951 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:36721
I20260619 10:29:18.458341 26018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:36721 every 8 connection(s)
I20260619 10:29:18.458683 25951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:18.460011 26019 sys_catalog.cc:263] Verifying existing consensus state
I20260619 10:29:18.460582 26019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Bootstrap starting.
I20260619 10:29:18.468382 26019 log.cc:824] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:18.468406 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 25951
I20260619 10:29:18.468699 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:34201
--local_ip_for_outbound_sockets=127.9.90.65
--tserver_master_addrs=127.9.90.126:36721
--webserver_port=46057
--webserver_interface=127.9.90.65
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:29:18.470551 26019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:29:18.470815 26019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: Bootstrap complete.
I20260619 10:29:18.472806 26019 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:29:18.473012 26019 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d6d71cc0cc4fbdaee619f58138d65d, State: Initialized, Role: FOLLOWER
I20260619 10:29:18.473158 26019 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:29:18.473230 26019 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:18.473277 26019 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:18.473325 26019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 2 FOLLOWER]: Advancing to term 3
I20260619 10:29:18.474311 26019 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:29:18.474404 26019 leader_election.cc:302] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d6d71cc0cc4fbdaee619f58138d65d; no voters:
I20260619 10:29:18.474545 26019 leader_election.cc:288] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 10:29:18.474601 26023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 3 FOLLOWER]: Leader election won for term 3
I20260619 10:29:18.474776 26019 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:18.474805 26023 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [term 3 LEADER]: Becoming Leader. State: Replica: b2d6d71cc0cc4fbdaee619f58138d65d, State: Running, Role: LEADER
I20260619 10:29:18.474897 26023 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } }
I20260619 10:29:18.475250 26025 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d6d71cc0cc4fbdaee619f58138d65d. Latest consensus state: current_term: 3 leader_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } } }
I20260619 10:29:18.475235 26024 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d6d71cc0cc4fbdaee619f58138d65d" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36721 } } }
I20260619 10:29:18.475353 26025 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:18.475385 26024 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:18.476869 26039 catalog_manager.cc:1259] Loaded cluster ID: 99e0305724d2431bad8a7172d27c4956
I20260619 10:29:18.476919 26039 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: loading cluster ID for follower catalog manager: success
I20260619 10:29:18.478140 26039 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2d6d71cc0cc4fbdaee619f58138d65d: acquiring CA information for follower catalog manager: success
F20260619 10:29:18.478514 26039 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff4431389ad google::LogMessage::Fail() at ??:0
@ 0x7ff44313b88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff4431384de google::LogMessage::Flush() at ??:0
@ 0x7ff443139a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff4458a5104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff44b0916b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7ff44b091989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7ff44b091e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7ff44b0948c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7ff444ffb954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff445e5f6db start_thread at ??:0
@ 0x7ff441c3661f clone at ??:0
W20260619 10:29:18.576094 26022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:18.576355 26022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:18.576387 26022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:18.578055 26022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:18.578112 26022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:18.578150 26022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:29:18.580214 26022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:34201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=46057
--tserver_master_addrs=127.9.90.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:18.580464 26022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:18.580724 26022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:18.585876 26057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:18.585879 26060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:18.586081 26022 server_base.cc:1034] running on GCE node
W20260619 10:29:18.586091 26058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:18.586300 26022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:18.586536 26022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:18.587677 26022 hybrid_clock.cc:648] HybridClock initialized: now 1781864958587664 us; error 30 us; skew 500 ppm
Jun 19 10:29:18 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:18.589936 26022 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:29:18.590389 26022 webserver.cc:460] Webserver started at http://127.9.90.65:46057/ using document root <none> and password file <none>
I20260619 10:29:18.590598 26022 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:18.590641 26022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:18.591929 26022 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:29:18.592625 26067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.592818 26022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:29:18.592957 26022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "ad281fd2bdae498180ad14ee0f9a8a70"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "c00d52d4441aced37465fca6ecd99b7b"
server_key_iv: "e9cd184224bd228b302c4052c15c97b0"
server_key_version: "encryptionkey@0"
I20260619 10:29:18.593274 26022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:18.625661 26022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:18.627795 26022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:18.627931 26022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:18.628140 26022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:18.628579 26074 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 10:29:18.630515 26022 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 10:29:18.630563 26022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 10:29:18.630605 26022 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 10:29:18.631202 26022 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 10:29:18.631235 26022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 10:29:18.631328 26074 tablet_bootstrap.cc:492] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Bootstrap starting.
I20260619 10:29:18.637818 26022 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:34201
I20260619 10:29:18.637954 26181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:34201 every 8 connection(s)
I20260619 10:29:18.638207 26022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:29:18.642061 26074 log.cc:824] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:18.643693 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 26022
I20260619 10:29:18.643947 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:41175
--local_ip_for_outbound_sockets=127.9.90.66
--tserver_master_addrs=127.9.90.126:36721
--webserver_port=43777
--webserver_interface=127.9.90.66
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:29:18.644678 26074 tablet_bootstrap.cc:492] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 10:29:18.644973 26074 tablet_bootstrap.cc:492] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Bootstrap complete.
I20260619 10:29:18.646080 26074 ts_tablet_manager.cc:1397] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.008s
I20260619 10:29:18.647478 26074 raft_consensus.cc:348] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:29:18.647818 26074 raft_consensus.cc:729] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad281fd2bdae498180ad14ee0f9a8a70, State: Initialized, Role: FOLLOWER
I20260619 10:29:18.647986 26074 consensus_queue.cc:260] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:29:18.648272 26074 ts_tablet_manager.cc:1428] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260619 10:29:18.679247 26184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:29:18.638893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:60775)
0619 10:29:18.639058 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:18.639107 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:18.639526 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:18.679002 (+ 39476us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
W20260619 10:29:18.679543 26182 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:36721 (0 consecutive failures): Network error: Failed to ping master at 127.9.90.126:36721: Client connection negotiation failed: client connection to 127.9.90.126:36721: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 10:29:18.679890 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:18.679776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:36453)
0619 10:29:18.679828 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:18.679867 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:29:18.680230 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:18.680154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:53765)
0619 10:29:18.680190 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:18.680212 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 10:29:18.680318 26182 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 10:29:18.723097 26186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:18.723289 26186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:18.723309 26186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:18.724370 26186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:18.724399 26186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:18.724424 26186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:29:18.725807 26186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:41175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=43777
--tserver_master_addrs=127.9.90.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:18.725975 26186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:18.726154 26186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:18.730659 26195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:18.730814 26193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:18.730655 26192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:18.730842 26186 server_base.cc:1034] running on GCE node
I20260619 10:29:18.731168 26186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:18.731374 26186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:18.732508 26186 hybrid_clock.cc:648] HybridClock initialized: now 1781864958732494 us; error 27 us; skew 500 ppm
Jun 19 10:29:18 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:18.734808 26186 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:29:18.735209 26186 webserver.cc:460] Webserver started at http://127.9.90.66:43777/ using document root <none> and password file <none>
I20260619 10:29:18.735395 26186 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:18.735441 26186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:18.736626 26186 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:29:18.737118 26202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.737260 26186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.737334 26186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "307ad1c5818f47aaaafb7531c14cef84"
format_stamp: "Formatted at 2026-06-19 10:28:36 on dist-test-slave-hw7q"
server_key: "479ad63b23533329a77613ddd9db9d41"
server_key_iv: "dc130000a8029c95edb506264a2c3026"
server_key_version: "encryptionkey@0"
I20260619 10:29:18.737618 26186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:18.767648 26186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:18.769701 26186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:18.769855 26186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:18.770033 26186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:18.770294 26186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:18.770326 26186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.770354 26186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:18.770377 26186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.775540 26186 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:41175
I20260619 10:29:18.775604 26315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:41175 every 8 connection(s)
I20260619 10:29:18.775902 26186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:29:18.776846 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:18.776514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:54257)
0619 10:29:18.776643 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:18.776707 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
W20260619 10:29:18.777204 26316 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:36721 (0 consecutive failures): Network error: Failed to ping master at 127.9.90.126:36721: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
I20260619 10:29:18.777451 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:18.777374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:52629)
0619 10:29:18.777410 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:18.777434 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 10:29:18.777750 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:18.777685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:44145)
0619 10:29:18.777716 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:18.777736 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260619 10:29:18.777855 26316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 10:29:18.778630 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 26186
I20260619 10:29:18.778885 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:42495
--local_ip_for_outbound_sockets=127.9.90.67
--tserver_master_addrs=127.9.90.126:36721
--webserver_port=40059
--webserver_interface=127.9.90.67
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:18.855195 26319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:18.855388 26319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:18.855409 26319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:18.856451 26319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:18.856482 26319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:18.856504 26319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:29:18.857833 26319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:42495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=40059
--tserver_master_addrs=127.9.90.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:18.857991 26319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:18.858140 26319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:18.862653 26325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:18.862793 26319 server_base.cc:1034] running on GCE node
W20260619 10:29:18.862654 26327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:18.862665 26324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:18.863051 26319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:18.863260 26319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:18.864394 26319 hybrid_clock.cc:648] HybridClock initialized: now 1781864958864385 us; error 29 us; skew 500 ppm
Jun 19 10:29:18 dist-test-slave-hw7q krb5kdc[24877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:18.866693 26319 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:29:18.867108 26319 webserver.cc:460] Webserver started at http://127.9.90.67:40059/ using document root <none> and password file <none>
I20260619 10:29:18.867319 26319 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:18.867367 26319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:18.868556 26319 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:29:18.869112 26334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.869273 26319 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.869352 26319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "ecebd4ffb75d456fb997e8cea6beb41e"
format_stamp: "Formatted at 2026-06-19 10:28:37 on dist-test-slave-hw7q"
server_key: "78ed14e89e4747ffc6b3d2aabea03943"
server_key_iv: "2cce6635830ea1dd2b567d6549d564e2"
server_key_version: "encryptionkey@0"
I20260619 10:29:18.869645 26319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:18.893573 26319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:18.895671 26319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:18.895805 26319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:18.896034 26319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:18.896363 26319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:18.896395 26319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.896425 26319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:18.896441 26319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:18.901855 26319 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:42495
I20260619 10:29:18.901906 26447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:42495 every 8 connection(s)
I20260619 10:29:18.902230 26319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:29:18.903250 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:18.902838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:57429)
0619 10:29:18.903010 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:18.903088 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260619 10:29:18.903278 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 26319
W20260619 10:29:18.903704 26448 heartbeater.cc:644] Failed to heartbeat to 127.9.90.126:36721 (0 consecutive failures): Network error: Failed to ping master at 127.9.90.126:36721: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
I20260619 10:29:18.903980 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:18.903893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:43089)
0619 10:29:18.903932 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:18.903958 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 10:29:18.904222 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:18.904153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:53161)
0619 10:29:18.904188 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:18.904205 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 10:29:18.904314 26448 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 10:29:19.680977 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:19.680702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:58981)
0619 10:29:19.680866 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:19.680916 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260619 10:29:19.778456 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:19.778201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:56555)
0619 10:29:19.778364 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:19.778405 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260619 10:29:19.904807 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:19.904608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:53639)
0619 10:29:19.904709 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:19.904760 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260619 10:29:20.349140 26455 raft_consensus.cc:482] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:20.349265 26455 raft_consensus.cc:504] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:29:20.349539 26455 leader_election.cc:288] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175), ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495)
Jun 19 10:29:20 dist-test-slave-hw7q krb5kdc[24877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.66@KRBTEST.COM
Jun 19 10:29:20 dist-test-slave-hw7q krb5kdc[24877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.67@KRBTEST.COM
I20260619 10:29:20.362293 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:20.349826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:39183 (local address 127.9.90.66:41175)
0619 10:29:20.349981 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:20.349993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:20.350020 (+ 27us) server_negotiation.cc:407] Connection header received
0619 10:29:20.350491 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:20.350494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:20.350538 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:20.350669 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:20.351410 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:20.352294 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:20.353240 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:20.353608 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:20.353701 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:20.359028 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:20.359032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:20.359032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:20.359042 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:20.360610 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:20.361186 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:20.361189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:20.361189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:20.361274 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:20.361484 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:20.361485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:20.361485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:20.361702 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:20.361804 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:20.362058 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 10:29:20.362171 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":49,"threads_started":1}
I20260619 10:29:20.362282 26457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:20.349735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:41175 (local address 127.9.90.65:39183)
0619 10:29:20.349932 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:20.349943 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:20.350014 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:20.350720 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:20.350723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:20.351106 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:20.351301 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:20.351306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:20.352379 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:20.352380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:20.353154 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:20.353156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:20.353682 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:20.353683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:20.353845 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:20.357474 (+ 3629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:20.357478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:20.358926 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:20.360682 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:20.360687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:20.360689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:20.360918 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:20.361334 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:20.361335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:20.361335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:20.361431 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:20.361767 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:20.361772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:20.361917 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 10:29:20.362001 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 10:29:20.362174 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":49,"thread_start_us":98,"threads_started":1}
I20260619 10:29:20.362291 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:20.349920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:54949 (local address 127.9.90.67:42495)
0619 10:29:20.350071 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:20.350081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:20.350114 (+ 33us) server_negotiation.cc:407] Connection header received
0619 10:29:20.350526 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:20.350528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:20.350564 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:20.350700 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:20.351415 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:20.352272 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:20.353241 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:20.353621 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:20.353709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:20.359384 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:20.359388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:20.359389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:20.359400 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:20.360914 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:20.361200 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:20.361203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:20.361203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:20.361290 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:20.361495 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:20.361496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:20.361496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:20.361724 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:20.361814 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:20.362062 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 10:29:20.362178 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":55,"threads_started":1}
I20260619 10:29:20.362381 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:20.349735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:42495 (local address 127.9.90.65:54949)
0619 10:29:20.349898 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:20.349910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:20.349967 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:20.350727 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:20.350728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:20.351106 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:20.351301 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:20.351306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:20.352363 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:20.352364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:20.353157 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:20.353159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:20.353690 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:20.353691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:20.353844 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:20.357462 (+ 3618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:20.357472 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:20.359281 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:20.360981 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:20.360981 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:20.360982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:20.361115 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:20.361377 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:20.361378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:20.361378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:20.361443 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:20.361809 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:20.361810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:20.361917 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 10:29:20.362002 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 10:29:20.362273 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":203,"thread_start_us":112,"threads_started":1}
W20260619 10:29:20.363058 26069 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
W20260619 10:29:20.363389 26070 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
I20260619 10:29:20.363430 26070 leader_election.cc:302] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad281fd2bdae498180ad14ee0f9a8a70; no voters: 307ad1c5818f47aaaafb7531c14cef84, ecebd4ffb75d456fb997e8cea6beb41e
I20260619 10:29:20.363523 26455 raft_consensus.cc:2726] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:29:20.681579 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:20.681438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:45959)
0619 10:29:20.681503 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:20.681541 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 10:29:20.779058 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:20.778846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:55637)
0619 10:29:20.778971 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:20.779013 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260619 10:29:20.905376 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:20.905179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:44711)
0619 10:29:20.905306 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:20.905341 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260619 10:29:21.682260 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:21.682000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:42841)
0619 10:29:21.682156 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:21.682197 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260619 10:29:21.779631 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:21.779451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:56703)
0619 10:29:21.779557 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:21.779593 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260619 10:29:21.906026 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:21.905837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:34663)
0619 10:29:21.905934 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:21.905975 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":57,"threads_started":1}
I20260619 10:29:22.045974 26465 raft_consensus.cc:482] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:22.046066 26465 raft_consensus.cc:504] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:29:22.046262 26465 leader_election.cc:288] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175), ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495)
W20260619 10:29:22.046679 26070 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
W20260619 10:29:22.046792 26069 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
I20260619 10:29:22.046840 26069 leader_election.cc:302] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad281fd2bdae498180ad14ee0f9a8a70; no voters: 307ad1c5818f47aaaafb7531c14cef84, ecebd4ffb75d456fb997e8cea6beb41e
I20260619 10:29:22.046908 26465 raft_consensus.cc:2726] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:29:22.682931 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:22.682706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:36865)
0619 10:29:22.682846 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:22.682885 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260619 10:29:22.780256 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:22.780048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:40413)
0619 10:29:22.780163 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:22.780220 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260619 10:29:22.906638 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:22.906424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:54423)
0619 10:29:22.906560 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:22.906598 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260619 10:29:23.683681 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:23.683437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:37673)
0619 10:29:23.683592 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:23.683635 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260619 10:29:23.780905 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:23.780691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:35305)
0619 10:29:23.780820 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:23.780857 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260619 10:29:23.907295 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:23.907069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:40611)
0619 10:29:23.907191 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:23.907254 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260619 10:29:24.684366 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:24.684112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:48171)
0619 10:29:24.684271 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:24.684316 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":59,"threads_started":1}
I20260619 10:29:24.781591 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:24.781339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:36589)
0619 10:29:24.781501 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:24.781545 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260619 10:29:24.835609 26474 raft_consensus.cc:482] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:24.835701 26474 raft_consensus.cc:504] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:29:24.835867 26474 leader_election.cc:288] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175), ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495)
W20260619 10:29:24.836216 26070 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
W20260619 10:29:24.836298 26069 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
I20260619 10:29:24.836341 26069 leader_election.cc:302] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad281fd2bdae498180ad14ee0f9a8a70; no voters: 307ad1c5818f47aaaafb7531c14cef84, ecebd4ffb75d456fb997e8cea6beb41e
I20260619 10:29:24.836431 26474 raft_consensus.cc:2726] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:29:24.907950 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:24.907728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:51795)
0619 10:29:24.907867 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:24.907906 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260619 10:29:25.685139 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:25.684912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:50667)
0619 10:29:25.685053 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:25.685091 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260619 10:29:25.782200 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:25.782016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:49943)
0619 10:29:25.782124 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:25.782162 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260619 10:29:25.908582 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:25.908368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:48247)
0619 10:29:25.908505 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:25.908541 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":56,"threads_started":1}
I20260619 10:29:26.685844 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:26.685588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:33123)
0619 10:29:26.685710 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:26.685765 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260619 10:29:26.782820 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:26.782571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:52399)
0619 10:29:26.782731 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:26.782773 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":55,"threads_started":1}
I20260619 10:29:26.909216 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:26.908980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:32875)
0619 10:29:26.909108 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:26.909173 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260619 10:29:27.686571 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:27.686332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:60885)
0619 10:29:27.686485 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:27.686524 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260619 10:29:27.783469 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:27.783245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:42795)
0619 10:29:27.783385 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:27.783424 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260619 10:29:27.909861 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:27.909640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:58333)
0619 10:29:27.909746 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:27.909782 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260619 10:29:28.687266 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:28.687048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:39621)
0619 10:29:28.687166 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:28.687198 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260619 10:29:28.784107 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:28.783852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:36087)
0619 10:29:28.783988 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:28.784036 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260619 10:29:28.910477 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:28.910248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:35841)
0619 10:29:28.910394 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:28.910433 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260619 10:29:29.319407 26488 raft_consensus.cc:482] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:29.319504 26488 raft_consensus.cc:504] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:29:29.319643 26488 leader_election.cc:288] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175), ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495)
W20260619 10:29:29.320015 26069 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
W20260619 10:29:29.320104 26070 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
I20260619 10:29:29.320165 26070 leader_election.cc:302] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad281fd2bdae498180ad14ee0f9a8a70; no voters: 307ad1c5818f47aaaafb7531c14cef84, ecebd4ffb75d456fb997e8cea6beb41e
I20260619 10:29:29.320276 26488 raft_consensus.cc:2726] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:29:29.687973 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:29.687730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:39267)
0619 10:29:29.687882 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:29.687926 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260619 10:29:29.784911 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:29.784692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:36603)
0619 10:29:29.784831 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:29.784868 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":56,"threads_started":1}
I20260619 10:29:29.911063 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:29.910882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:38869)
0619 10:29:29.910992 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:29.911025 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260619 10:29:30.688773 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:30.688515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:38721)
0619 10:29:30.688670 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:30.688720 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260619 10:29:30.785521 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:30.785337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:46433)
0619 10:29:30.785449 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:30.785481 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260619 10:29:30.911646 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:30.911432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:54485)
0619 10:29:30.911574 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:30.911608 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260619 10:29:31.689514 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:31.689278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:42669)
0619 10:29:31.689426 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:31.689466 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260619 10:29:31.786123 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:31.785922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:37927)
0619 10:29:31.786051 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:31.786083 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260619 10:29:31.912312 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:31.912067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:58743)
0619 10:29:31.912231 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:31.912269 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260619 10:29:32.608915 26498 raft_consensus.cc:482] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:32.609035 26498 raft_consensus.cc:504] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307ad1c5818f47aaaafb7531c14cef84" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 41175 } } peers { permanent_uuid: "ecebd4ffb75d456fb997e8cea6beb41e" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 42495 } } peers { permanent_uuid: "ad281fd2bdae498180ad14ee0f9a8a70" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 34201 } }
I20260619 10:29:32.609200 26498 leader_election.cc:288] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175), ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495)
W20260619 10:29:32.609555 26070 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 307ad1c5818f47aaaafb7531c14cef84 (127.9.90.66:41175): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
W20260619 10:29:32.609633 26069 leader_election.cc:341] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ecebd4ffb75d456fb997e8cea6beb41e (127.9.90.67:42495): Not found: Tablet not found: 6aae1067db794d438fbe36d7164459e2
I20260619 10:29:32.609685 26069 leader_election.cc:302] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad281fd2bdae498180ad14ee0f9a8a70; no voters: 307ad1c5818f47aaaafb7531c14cef84, ecebd4ffb75d456fb997e8cea6beb41e
I20260619 10:29:32.609751 26498 raft_consensus.cc:2726] T 6aae1067db794d438fbe36d7164459e2 P ad281fd2bdae498180ad14ee0f9a8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 10:29:32.690251 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:32.690011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:59833)
0619 10:29:32.690158 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:32.690203 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260619 10:29:32.786782 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:32.786565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:39023)
0619 10:29:32.786688 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:32.786736 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260619 10:29:32.912930 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:32.912704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.67:32999)
0619 10:29:32.912856 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:32.912885 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260619 10:29:33.690965 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:33.690736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.65:49681)
0619 10:29:33.690879 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:33.690919 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260619 10:29:33.787437 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:33.787245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36721 (local address 127.9.90.66:35141)
0619 10:29:33.787368 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:33.787398 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.126:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260619 10:29:33.903751 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 26022
I20260619 10:29:33.907817 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 26186
I20260619 10:29:33.911345 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 26319
2026-06-19T10:29:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58493 ms)
[----------] 30 tests from SecurityITest (219718 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:33 dist-test-slave-hw7q krb5kdc[26506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:33 dist-test-slave-hw7q krb5kdc[26506](info): set up 2 sockets
Jun 19 10:29:33 dist-test-slave-hw7q krb5kdc[26506](info): commencing operation
krb5kdc: starting...
W20260619 10:29:34.857194 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:34 dist-test-slave-hw7q krb5kdc[26506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:29:34.962095 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:38383
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:38383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:35.042439 26519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:35.042635 26519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:35.042656 26519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:35.043745 26519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:35.043780 26519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:35.043793 26519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:35.043804 26519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:35.043815 26519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:35.045209 26519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:38383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:35.045397 26519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:35.045583 26519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:35.050081 26525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.050109 26519 server_base.cc:1034] running on GCE node
W20260619 10:29:35.050134 26527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:35.050259 26524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.050546 26519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:35.050789 26519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:35.051931 26519 hybrid_clock.cc:648] HybridClock initialized: now 1781864975051915 us; error 29 us; skew 500 ppm
Jun 19 10:29:35 dist-test-slave-hw7q krb5kdc[26506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864975, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:35.054272 26519 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:29:35.054672 26519 webserver.cc:460] Webserver started at http://127.9.90.126:36523/ using document root <none> and password file <none>
I20260619 10:29:35.054880 26519 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:35.054922 26519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:35.055044 26519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:35.056147 26519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "03e1458851d24fc7be8449eb70390973"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "d1abaae7495c361f542a07e6b18905f6"
server_key_iv: "79f969e9444adc775e9685a6e7691e3a"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.056464 26519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "03e1458851d24fc7be8449eb70390973"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "d1abaae7495c361f542a07e6b18905f6"
server_key_iv: "79f969e9444adc775e9685a6e7691e3a"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.057720 26519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:29:35.058386 26534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.058529 26519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.058595 26519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "03e1458851d24fc7be8449eb70390973"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "d1abaae7495c361f542a07e6b18905f6"
server_key_iv: "79f969e9444adc775e9685a6e7691e3a"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.058645 26519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:35.074715 26519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:35.076670 26519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:35.076773 26519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:35.080384 26519 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:38383
I20260619 10:29:35.080432 26586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:38383 every 8 connection(s)
I20260619 10:29:35.080745 26519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:35.081573 26587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:35.083698 26587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973: Bootstrap starting.
I20260619 10:29:35.084196 26587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:35.084419 26587 log.cc:824] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:35.085026 26587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973: No bootstrap required, opened a new log
I20260619 10:29:35.086164 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 26519
I20260619 10:29:35.086272 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:35.086462 9577 external_mini_cluster.cc:1428] Setting key fb8180cd63761c357e002dcc9ba32fdc
I20260619 10:29:35.086874 26587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e1458851d24fc7be8449eb70390973" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38383 } }
I20260619 10:29:35.087049 26587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:35.087134 26587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e1458851d24fc7be8449eb70390973, State: Initialized, Role: FOLLOWER
I20260619 10:29:35.087270 26587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e1458851d24fc7be8449eb70390973" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38383 } }
I20260619 10:29:35.087347 26587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:35.087378 26587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:35.087422 26587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:35.088279 26587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e1458851d24fc7be8449eb70390973" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38383 } }
I20260619 10:29:35.088382 26587 leader_election.cc:302] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03e1458851d24fc7be8449eb70390973; no voters:
I20260619 10:29:35.088522 26587 leader_election.cc:288] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:35.088552 26592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:35.088721 26592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [term 1 LEADER]: Becoming Leader. State: Replica: 03e1458851d24fc7be8449eb70390973, State: Running, Role: LEADER
I20260619 10:29:35.088752 26587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:35.088855 26592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e1458851d24fc7be8449eb70390973" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38383 } }
I20260619 10:29:35.089653 26593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e1458851d24fc7be8449eb70390973. Latest consensus state: current_term: 1 leader_uuid: "03e1458851d24fc7be8449eb70390973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e1458851d24fc7be8449eb70390973" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38383 } } }
I20260619 10:29:35.089738 26593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:35.089952 26605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:35.089986 26594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03e1458851d24fc7be8449eb70390973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e1458851d24fc7be8449eb70390973" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 38383 } } }
I20260619 10:29:35.090073 26594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:35.090515 26605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:35.091957 26605 catalog_manager.cc:1347] Generated new cluster ID: 9f7f5bbd02c142dd91f929c7b5834199
I20260619 10:29:35.092005 26605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:29:35 dist-test-slave-hw7q krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:35.097319 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.086949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.9.90.126:38383)
0619 10:29:35.087141 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.087150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.087181 (+ 31us) server_negotiation.cc:407] Connection header received
0619 10:29:35.087293 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.087299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.087340 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.087460 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:35.088131 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.089361 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.090448 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.090998 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.091105 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.093652 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:35.093656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:35.093656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:35.093666 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:35.095485 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.095916 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.095918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.095919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.096016 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.096225 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.096226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.096226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.096481 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:35.096961 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.097023 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.097140 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260619 10:29:35.143357 26605 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:35.144081 26605 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:35.150614 26605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973: Generated new TSK 0
I20260619 10:29:35.150805 26605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:29:35.177721 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:38383
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:35.271037 26613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:35.271231 26613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:35.271262 26613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:35.272330 26613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:35.272368 26613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:35.272401 26613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:29:35.273743 26613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:35.273970 26613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:35.274160 26613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:35.278663 26618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:35.278674 26619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.278841 26613 server_base.cc:1034] running on GCE node
W20260619 10:29:35.278672 26621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.279110 26613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:35.279323 26613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:35.280450 26613 hybrid_clock.cc:648] HybridClock initialized: now 1781864975280438 us; error 30 us; skew 500 ppm
Jun 19 10:29:35 dist-test-slave-hw7q krb5kdc[26506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864975, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:35.282778 26613 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:29:35.283150 26613 webserver.cc:460] Webserver started at http://127.9.90.65:39919/ using document root <none> and password file <none>
I20260619 10:29:35.283331 26613 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:35.283375 26613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:35.283493 26613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:35.284466 26613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "e97e2328572445a8a0ea6b285b518795"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "fa41bc8f525c658845c170d776df5787"
server_key_iv: "468fd1becf02536119311647eba6a86e"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.284786 26613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "e97e2328572445a8a0ea6b285b518795"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "fa41bc8f525c658845c170d776df5787"
server_key_iv: "468fd1becf02536119311647eba6a86e"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.285935 26613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:29:35.286620 26628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.286758 26613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:35.286839 26613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "e97e2328572445a8a0ea6b285b518795"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "fa41bc8f525c658845c170d776df5787"
server_key_iv: "468fd1becf02536119311647eba6a86e"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.286901 26613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:35.298854 26613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:35.300452 26613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:35.300581 26613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:35.300794 26613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:35.301061 26613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:35.301093 26613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.301119 26613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:35.301142 26613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.306567 26613 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:44269
I20260619 10:29:35.306610 26741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:44269 every 8 connection(s)
I20260619 10:29:35.306908 26613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:29:35.312305 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 26613
I20260619 10:29:35.312390 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
Jun 19 10:29:35 dist-test-slave-hw7q krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864975, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:35.312568 9577 external_mini_cluster.cc:1428] Setting key d06b96a578764fa26feb5afd5cf57dad
I20260619 10:29:35.315457 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.307530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:44777 (local address 127.9.90.126:38383)
0619 10:29:35.307581 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.307583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.307770 (+ 187us) server_negotiation.cc:407] Connection header received
0619 10:29:35.308268 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.308269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.308281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.308326 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:35.309231 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.309739 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.310700 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.310906 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.310961 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.312926 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:35.312927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:35.312927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:35.312935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:35.314281 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.314629 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.314630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.314631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.314682 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.314888 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.314889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.314890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.314996 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:35.315051 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.315289 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.315372 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:29:35.315616 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.307492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:38383 (local address 127.9.90.65:44777)
0619 10:29:35.307647 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:35.307689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:35.308162 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:35.308395 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:35.308400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:35.308950 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:35.309134 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.309139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.309836 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.309837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.310621 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.310623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.310972 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.310973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.311095 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.311501 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:35.311509 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:35.312827 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:35.314349 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:35.314355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:35.314357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:35.314557 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:35.314741 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:35.314742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:35.314742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:35.314834 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:35.315050 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:35.315056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:35.315175 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 10:29:35.315262 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 10:29:35.315408 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260619 10:29:35.316010 26742 heartbeater.cc:347] Connected to a master server at 127.9.90.126:38383
I20260619 10:29:35.316099 26742 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:35.316263 26742 heartbeater.cc:511] Master 127.9.90.126:38383 requested a full tablet report, sending...
I20260619 10:29:35.316744 26551 ts_manager.cc:194] Registered new tserver with Master: e97e2328572445a8a0ea6b285b518795 (127.9.90.65:44269)
I20260619 10:29:35.317368 26551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:44777
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:29:35.333986 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:38383
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:35.410297 26747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:35.410482 26747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:35.410511 26747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:35.411609 26747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:35.411651 26747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:35.411679 26747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:29:35.413048 26747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:35.413254 26747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:35.413441 26747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:35.417899 26752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.418027 26747 server_base.cc:1034] running on GCE node
W20260619 10:29:35.417898 26753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:35.417923 26755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.418260 26747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:35.418476 26747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:35.419637 26747 hybrid_clock.cc:648] HybridClock initialized: now 1781864975419622 us; error 30 us; skew 500 ppm
Jun 19 10:29:35 dist-test-slave-hw7q krb5kdc[26506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864975, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:35.422027 26747 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:29:35.422397 26747 webserver.cc:460] Webserver started at http://127.9.90.66:38401/ using document root <none> and password file <none>
I20260619 10:29:35.422581 26747 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:35.422622 26747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:35.422715 26747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:35.423658 26747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "d71c1480d11d4c59bde951957fa83d71"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "8a6b53510d346ea778e8d5a34368a3bf"
server_key_iv: "e559a417f0d98118fff145ef4e379f81"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.423949 26747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "d71c1480d11d4c59bde951957fa83d71"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "8a6b53510d346ea778e8d5a34368a3bf"
server_key_iv: "e559a417f0d98118fff145ef4e379f81"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.425094 26747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:29:35.425694 26762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.425858 26747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.425925 26747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "d71c1480d11d4c59bde951957fa83d71"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "8a6b53510d346ea778e8d5a34368a3bf"
server_key_iv: "e559a417f0d98118fff145ef4e379f81"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.425990 26747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:35.468989 26747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:35.471571 26747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:35.471696 26747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:35.471909 26747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:35.472203 26747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:35.472234 26747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.472263 26747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:35.472286 26747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.477828 26747 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:40223
I20260619 10:29:35.477869 26875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:40223 every 8 connection(s)
I20260619 10:29:35.478184 26747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:29:35 dist-test-slave-hw7q krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864975, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:35.486145 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.478779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:49359 (local address 127.9.90.126:38383)
0619 10:29:35.478859 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.478861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.479030 (+ 169us) server_negotiation.cc:407] Connection header received
0619 10:29:35.479490 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.479491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.479500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.479545 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:35.480387 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.480872 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.481767 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.481983 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.482029 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.483931 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:35.483932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:35.483932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:35.483938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:35.485082 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.485397 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.485398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.485398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.485441 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.485620 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.485620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.485621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.485725 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:35.485773 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.486020 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.486082 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 10:29:35.486325 26878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.478764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:38383 (local address 127.9.90.66:49359)
0619 10:29:35.478912 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:35.478948 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:35.479418 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:35.479608 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:35.479611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:35.480137 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:35.480322 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.480326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.480930 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.480931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.481701 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.481704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.482045 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.482045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.482149 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.482536 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:35.482545 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:35.483849 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:35.485140 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:35.485143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:35.485144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:35.485347 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:35.485484 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:35.485485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:35.485485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:35.485569 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:35.485769 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:35.485771 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:35.485921 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 10:29:35.486003 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 10:29:35.486113 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260619 10:29:35.486743 26876 heartbeater.cc:347] Connected to a master server at 127.9.90.126:38383
I20260619 10:29:35.486828 26876 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:35.486989 26876 heartbeater.cc:511] Master 127.9.90.126:38383 requested a full tablet report, sending...
I20260619 10:29:35.487325 26551 ts_manager.cc:194] Registered new tserver with Master: d71c1480d11d4c59bde951957fa83d71 (127.9.90.66:40223)
I20260619 10:29:35.487666 26551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:49359
I20260619 10:29:35.488229 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 26747
I20260619 10:29:35.488335 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:29:35.488503 9577 external_mini_cluster.cc:1428] Setting key a041797b271e448d52c2ff8969428995
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:29:35.508164 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:38383
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:35.585335 26881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:35.585525 26881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:35.585546 26881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:35.586673 26881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:35.586711 26881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:35.586735 26881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:29:35.588081 26881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:35.588289 26881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:35.588476 26881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:35.593212 26886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:35.593238 26889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:35.593379 26887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.593662 26881 server_base.cc:1034] running on GCE node
I20260619 10:29:35.593878 26881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:35.594081 26881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:35.595219 26881 hybrid_clock.cc:648] HybridClock initialized: now 1781864975595205 us; error 28 us; skew 500 ppm
Jun 19 10:29:35 dist-test-slave-hw7q krb5kdc[26506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864975, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:35.597510 26881 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:29:35.597927 26881 webserver.cc:460] Webserver started at http://127.9.90.67:42955/ using document root <none> and password file <none>
I20260619 10:29:35.598140 26881 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:35.598186 26881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:35.598315 26881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:35.599280 26881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "fb21425031ec4851a8c3728b429f4925"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "8c7ddf6e5dd0c575ae465feb7b8615db"
server_key_iv: "5e9a35c8fe863bf36c8f224b6fc4a98e"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.599596 26881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "fb21425031ec4851a8c3728b429f4925"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "8c7ddf6e5dd0c575ae465feb7b8615db"
server_key_iv: "5e9a35c8fe863bf36c8f224b6fc4a98e"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.600744 26881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:29:35.601351 26896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.601497 26881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:35.601588 26881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "fb21425031ec4851a8c3728b429f4925"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "8c7ddf6e5dd0c575ae465feb7b8615db"
server_key_iv: "5e9a35c8fe863bf36c8f224b6fc4a98e"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.601655 26881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:35.610623 26881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:35.612178 26881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:35.612301 26881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:35.612478 26881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:35.612736 26881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:35.612768 26881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.612794 26881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:35.612818 26881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.618374 26881 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:38933
I20260619 10:29:35.618431 27009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:38933 every 8 connection(s)
I20260619 10:29:35.618685 26881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:29:35.621901 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 26881
I20260619 10:29:35.621978 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:29:35.622112 9577 external_mini_cluster.cc:1428] Setting key a657f54477faef5f846c75c151ac3ff1
Jun 19 10:29:35 dist-test-slave-hw7q krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864975, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:35.626673 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.619305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:46375 (local address 127.9.90.126:38383)
0619 10:29:35.619355 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.619357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.619532 (+ 175us) server_negotiation.cc:407] Connection header received
0619 10:29:35.619964 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.619965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.619973 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.620023 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:35.620856 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.621333 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.622454 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.622732 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.622788 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.624655 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:35.624657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:35.624657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:35.624664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:35.625650 (+ 986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.626016 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.626017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.626017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.626066 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.626226 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.626226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.626227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.626314 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:35.626357 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.626558 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.626613 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:29:35.626852 27012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.619277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:38383 (local address 127.9.90.67:46375)
0619 10:29:35.619420 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:35.619459 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:35.619897 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:35.620073 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:35.620076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:35.620600 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:35.620791 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.620795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.621395 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.621396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.622400 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.622402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.622804 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.622805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.622943 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.623321 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:35.623328 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:35.624578 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:35.625718 (+ 1140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:35.625723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:35.625724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:35.625960 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:35.626107 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:35.626107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:35.626107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:35.626186 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:35.626354 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:35.626357 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:35.626465 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 10:29:35.626536 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 10:29:35.626641 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260619 10:29:35.627251 27010 heartbeater.cc:347] Connected to a master server at 127.9.90.126:38383
I20260619 10:29:35.627328 27010 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:35.627480 27010 heartbeater.cc:511] Master 127.9.90.126:38383 requested a full tablet report, sending...
I20260619 10:29:35.627799 26551 ts_manager.cc:194] Registered new tserver with Master: fb21425031ec4851a8c3728b429f4925 (127.9.90.67:38933)
I20260619 10:29:35.628140 26551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:46375
I20260619 10:29:35.633486 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:29:35.639827 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.634769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.9.90.126:38383)
0619 10:29:35.634824 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.634825 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.634931 (+ 106us) server_negotiation.cc:407] Connection header received
0619 10:29:35.634985 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.634986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.634995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.635035 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:35.635568 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.636013 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.636692 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.636871 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.636920 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.637924 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:35.637925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:35.637926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:35.637932 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:35.638865 (+ 933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.639137 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.639138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.639139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.639185 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:35.639360 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:35.639361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:35.639361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:35.639491 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:35.639573 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.639674 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.639743 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:29:35.641057 26551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:29:35.642675 26551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:29:35.648943 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.644903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.9.90.67:38933)
0619 10:29:35.645341 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.645345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.645353 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:29:35.645386 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.645388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.645401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.645473 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:35.645612 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.646738 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.647866 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.648544 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.648619 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.648681 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.648820 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.648860 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":82,"threads_started":1}
I20260619 10:29:35.648939 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.644775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38933 (local address 127.0.0.1:35450)
0619 10:29:35.645082 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:35.645123 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:35.645191 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:35.645473 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:35.645474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:35.645476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:35.645534 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.645536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.646873 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.646874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.647796 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.647798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.648636 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.648637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.648710 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.648725 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 10:29:35.648800 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 10:29:35.648850 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":62,"threads_started":1}
I20260619 10:29:35.649094 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.644993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.9.90.65:44269)
0619 10:29:35.645217 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.645223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.645236 (+ 13us) server_negotiation.cc:407] Connection header received
0619 10:29:35.645275 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.645277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.645292 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.645362 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:35.645582 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.646749 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.647991 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.648665 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.648726 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.648764 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.648984 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.649030 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260619 10:29:35.648938 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.644783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:44269 (local address 127.0.0.1:57618)
0619 10:29:35.645118 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:35.645128 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:35.645192 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:35.645398 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:35.645401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:35.645403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:35.645473 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.645474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.646841 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.646842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.647918 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.647920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.648741 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.648741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.648793 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.648796 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:35.648841 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:29:35.648869 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":85,"threads_started":1}
I20260619 10:29:35.649780 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.644971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:40223 (local address 127.0.0.1:46962)
0619 10:29:35.645272 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:35.645279 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:35.645319 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:35.645758 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:35.645759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:35.645761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:35.645833 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.645835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.646656 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.646657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.647910 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.647913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.649554 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.649555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.649613 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.649617 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:35.649662 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:29:35.649718 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":4,"thread_start_us":42,"threads_started":1}
I20260619 10:29:35.649930 26676 tablet_service.cc:1467] Processing CreateTablet for tablet ccb7ff480cc44e99b90b0093f9e62934 (DEFAULT_TABLE table=test-table [id=cf8641d989084264894c3baeca5428e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:35.650072 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.645081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.9.90.66:40223)
0619 10:29:35.645257 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.645262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.645316 (+ 54us) server_negotiation.cc:407] Connection header received
0619 10:29:35.645601 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.645603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.645619 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.645687 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:35.645889 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.646569 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.648796 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.649476 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.649537 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.649590 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.649954 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.650000 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
I20260619 10:29:35.650269 26676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb7ff480cc44e99b90b0093f9e62934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:35.650597 26944 tablet_service.cc:1467] Processing CreateTablet for tablet ccb7ff480cc44e99b90b0093f9e62934 (DEFAULT_TABLE table=test-table [id=cf8641d989084264894c3baeca5428e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:35.650782 26810 tablet_service.cc:1467] Processing CreateTablet for tablet ccb7ff480cc44e99b90b0093f9e62934 (DEFAULT_TABLE table=test-table [id=cf8641d989084264894c3baeca5428e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:35.650904 26944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb7ff480cc44e99b90b0093f9e62934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:35.651073 26810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb7ff480cc44e99b90b0093f9e62934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:35.653434 27024 tablet_bootstrap.cc:492] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: Bootstrap starting.
I20260619 10:29:35.653672 27025 tablet_bootstrap.cc:492] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795: Bootstrap starting.
I20260619 10:29:35.654043 27026 tablet_bootstrap.cc:492] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: Bootstrap starting.
I20260619 10:29:35.654307 27025 tablet_bootstrap.cc:654] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:35.654307 27024 tablet_bootstrap.cc:654] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:35.654619 27026 tablet_bootstrap.cc:654] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:35.654879 27025 log.cc:824] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:35.654879 27024 log.cc:824] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:35.655084 27026 log.cc:824] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:35.655589 27025 tablet_bootstrap.cc:492] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795: No bootstrap required, opened a new log
I20260619 10:29:35.655645 27025 ts_tablet_manager.cc:1397] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:35.655737 27026 tablet_bootstrap.cc:492] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: No bootstrap required, opened a new log
I20260619 10:29:35.655793 27026 ts_tablet_manager.cc:1397] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:35.657148 27025 raft_consensus.cc:348] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } }
I20260619 10:29:35.657176 27026 raft_consensus.cc:348] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } }
I20260619 10:29:35.657286 27025 raft_consensus.cc:374] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:35.657308 27025 raft_consensus.cc:729] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97e2328572445a8a0ea6b285b518795, State: Initialized, Role: FOLLOWER
I20260619 10:29:35.657302 27026 raft_consensus.cc:374] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:35.657334 27026 raft_consensus.cc:729] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71c1480d11d4c59bde951957fa83d71, State: Initialized, Role: FOLLOWER
I20260619 10:29:35.657441 27026 consensus_queue.cc:260] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } }
I20260619 10:29:35.657680 27024 tablet_bootstrap.cc:492] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: No bootstrap required, opened a new log
I20260619 10:29:35.657696 27026 ts_tablet_manager.cc:1428] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:35.657723 26876 heartbeater.cc:503] Master 127.9.90.126:38383 was elected leader, sending a full tablet report...
I20260619 10:29:35.657748 27024 ts_tablet_manager.cc:1397] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260619 10:29:35.659387 27024 raft_consensus.cc:348] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } }
I20260619 10:29:35.659511 27024 raft_consensus.cc:374] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:35.659541 27024 raft_consensus.cc:729] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb21425031ec4851a8c3728b429f4925, State: Initialized, Role: FOLLOWER
I20260619 10:29:35.659647 27024 consensus_queue.cc:260] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } }
I20260619 10:29:35.659860 27024 ts_tablet_manager.cc:1428] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:35.659890 27010 heartbeater.cc:503] Master 127.9.90.126:38383 was elected leader, sending a full tablet report...
I20260619 10:29:35.659966 27025 consensus_queue.cc:260] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } }
I20260619 10:29:35.660224 27025 ts_tablet_manager.cc:1428] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 10:29:35.660349 26742 heartbeater.cc:503] Master 127.9.90.126:38383 was elected leader, sending a full tablet report...
I20260619 10:29:35.707291 27032 raft_consensus.cc:482] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:35.707427 27032 raft_consensus.cc:504] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } }
I20260619 10:29:35.707787 27032 leader_election.cc:288] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb21425031ec4851a8c3728b429f4925 (127.9.90.67:38933), d71c1480d11d4c59bde951957fa83d71 (127.9.90.66:40223)
I20260619 10:29:35.710510 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.708006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:33425 (local address 127.9.90.67:38933)
0619 10:29:35.708067 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.708069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.708077 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:29:35.708104 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.708106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.708117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.708177 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:35.708439 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.708906 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.709820 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.710204 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.710263 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.710273 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.710408 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.710441 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:29:35.710510 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.707938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:38933 (local address 127.9.90.65:33425)
0619 10:29:35.707978 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:35.707989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:35.708027 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:35.708184 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:35.708186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:35.708187 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:35.708261 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.708262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.708976 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.708977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.709726 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.709728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.710273 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.710274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.710349 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.710352 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:35.710396 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 10:29:35.710436 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 10:29:35.710573 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.708013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:40147 (local address 127.9.90.66:40223)
0619 10:29:35.708058 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.708060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.708313 (+ 253us) server_negotiation.cc:407] Connection header received
0619 10:29:35.708352 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.708353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.708364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.708401 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:35.708540 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.708992 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.709916 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.710285 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.710343 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.710353 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.710478 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.710511 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:29:35.710762 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.707943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:40223 (local address 127.9.90.65:40147)
0619 10:29:35.708245 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:35.708253 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:35.708305 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:35.708413 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:35.708413 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:35.708414 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:35.708470 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.708471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.709063 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.709064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.709845 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:35.709847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.710355 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.710356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:35.710416 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.710420 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:35.710463 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:29:35.710498 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":129,"thread_start_us":93,"threads_started":1}
I20260619 10:29:35.710937 26964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb7ff480cc44e99b90b0093f9e62934" candidate_uuid: "e97e2328572445a8a0ea6b285b518795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb21425031ec4851a8c3728b429f4925" is_pre_election: true
I20260619 10:29:35.711093 26964 raft_consensus.cc:2454] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e97e2328572445a8a0ea6b285b518795 in term 0.
I20260619 10:29:35.711066 26830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb7ff480cc44e99b90b0093f9e62934" candidate_uuid: "e97e2328572445a8a0ea6b285b518795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71c1480d11d4c59bde951957fa83d71" is_pre_election: true
I20260619 10:29:35.711186 26830 raft_consensus.cc:2454] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e97e2328572445a8a0ea6b285b518795 in term 0.
I20260619 10:29:35.711305 26632 leader_election.cc:302] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e97e2328572445a8a0ea6b285b518795, fb21425031ec4851a8c3728b429f4925; no voters:
I20260619 10:29:35.711455 27032 raft_consensus.cc:2781] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:29:35.711505 27032 raft_consensus.cc:482] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:29:35.711521 27032 raft_consensus.cc:3037] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:35.712157 27032 raft_consensus.cc:504] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } }
I20260619 10:29:35.712261 27032 leader_election.cc:288] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [CANDIDATE]: Term 1 election: Requested vote from peers fb21425031ec4851a8c3728b429f4925 (127.9.90.67:38933), d71c1480d11d4c59bde951957fa83d71 (127.9.90.66:40223)
I20260619 10:29:35.712412 26830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb7ff480cc44e99b90b0093f9e62934" candidate_uuid: "e97e2328572445a8a0ea6b285b518795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71c1480d11d4c59bde951957fa83d71"
I20260619 10:29:35.712471 26964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb7ff480cc44e99b90b0093f9e62934" candidate_uuid: "e97e2328572445a8a0ea6b285b518795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb21425031ec4851a8c3728b429f4925"
I20260619 10:29:35.712491 26830 raft_consensus.cc:3037] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:35.712558 26964 raft_consensus.cc:3037] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:35.713205 26830 raft_consensus.cc:2454] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e97e2328572445a8a0ea6b285b518795 in term 1.
I20260619 10:29:35.713203 26964 raft_consensus.cc:2454] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e97e2328572445a8a0ea6b285b518795 in term 1.
I20260619 10:29:35.713371 26630 leader_election.cc:302] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d71c1480d11d4c59bde951957fa83d71, e97e2328572445a8a0ea6b285b518795; no voters:
I20260619 10:29:35.713469 27032 raft_consensus.cc:2781] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:35.713618 27032 raft_consensus.cc:686] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [term 1 LEADER]: Becoming Leader. State: Replica: e97e2328572445a8a0ea6b285b518795, State: Running, Role: LEADER
I20260619 10:29:35.713698 27032 consensus_queue.cc:237] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } }
I20260619 10:29:35.714504 26551 catalog_manager.cc:5495] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 reported cstate change: term changed from 0 to 1, leader changed from <none> to e97e2328572445a8a0ea6b285b518795 (127.9.90.65). New cstate: current_term: 1 leader_uuid: "e97e2328572445a8a0ea6b285b518795" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e2328572445a8a0ea6b285b518795" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 } health_report { overall_health: UNKNOWN } } }
I20260619 10:29:35.720651 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.718759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.9.90.65:44269)
0619 10:29:35.718797 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.718799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.718807 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:29:35.718861 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.718862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.718878 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.718929 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:29:35.719025 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.719457 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.720125 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.720297 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.720335 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.720439 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:29:35.720526 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:29:35.720575 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.720588 (+ 13us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.720602 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:29:35.722262 26830 raft_consensus.cc:1261] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [term 1 FOLLOWER]: Refusing update from remote peer e97e2328572445a8a0ea6b285b518795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:35.722271 26964 raft_consensus.cc:1261] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [term 1 FOLLOWER]: Refusing update from remote peer e97e2328572445a8a0ea6b285b518795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:35.722554 27032 consensus_queue.cc:1035] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d71c1480d11d4c59bde951957fa83d71" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:35.722714 27034 consensus_queue.cc:1035] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb21425031ec4851a8c3728b429f4925" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:35.725123 27038 mvcc.cc:205] Tried to move back new op lower bound from 7298518940555739136 to 7298518940523978752. Current Snapshot: MvccSnapshot[applied={T|T < 7298518940555739136}]
I20260619 10:29:35.727356 26551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:35.727437 26551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:35.728075 26551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03e1458851d24fc7be8449eb70390973: Sending DeleteTablet for 3 replicas of tablet ccb7ff480cc44e99b90b0093f9e62934
I20260619 10:29:35.728363 26944 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb7ff480cc44e99b90b0093f9e62934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:35 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:35450
I20260619 10:29:35.728389 26676 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb7ff480cc44e99b90b0093f9e62934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:35 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:57618
I20260619 10:29:35.728399 26810 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb7ff480cc44e99b90b0093f9e62934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:35 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:46962
I20260619 10:29:35.728639 27047 tablet_replica.cc:330] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: stopping tablet replica
I20260619 10:29:35.728789 27049 tablet_replica.cc:330] T ccb7ff480cc44e99b90b0093f9e62934 P e97e2328572445a8a0ea6b285b518795: stopping tablet replica
I20260619 10:29:35.728818 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 26613
I20260619 10:29:35.729321 27047 raft_consensus.cc:2229] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:29:35.729422 27047 raft_consensus.cc:2258] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:35.729595 27048 tablet_replica.cc:330] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: stopping tablet replica
I20260619 10:29:35.729676 27048 raft_consensus.cc:2229] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:29:35.729763 27048 raft_consensus.cc:2258] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:35.729948 27047 ts_tablet_manager.cc:1905] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:35.730238 27048 ts_tablet_manager.cc:1905] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:35.731467 27047 ts_tablet_manager.cc:1918] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:35.731539 27047 log.cc:1196] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/ccb7ff480cc44e99b90b0093f9e62934
I20260619 10:29:35.731652 27048 ts_tablet_manager.cc:1918] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:35.731717 27048 log.cc:1196] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/ccb7ff480cc44e99b90b0093f9e62934
I20260619 10:29:35.731835 27047 ts_tablet_manager.cc:1939] T ccb7ff480cc44e99b90b0093f9e62934 P fb21425031ec4851a8c3728b429f4925: Deleting consensus metadata
I20260619 10:29:35.731984 27048 ts_tablet_manager.cc:1939] T ccb7ff480cc44e99b90b0093f9e62934 P d71c1480d11d4c59bde951957fa83d71: Deleting consensus metadata
I20260619 10:29:35.732232 26538 catalog_manager.cc:4841] TS fb21425031ec4851a8c3728b429f4925 (127.9.90.67:38933): tablet ccb7ff480cc44e99b90b0093f9e62934 (table test-table [id=cf8641d989084264894c3baeca5428e6]) successfully deleted
I20260619 10:29:35.732311 26536 catalog_manager.cc:4841] TS d71c1480d11d4c59bde951957fa83d71 (127.9.90.66:40223): tablet ccb7ff480cc44e99b90b0093f9e62934 (table test-table [id=cf8641d989084264894c3baeca5428e6]) successfully deleted
W20260619 10:29:35.733927 26536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:44269 (error 108)
I20260619 10:29:35.734090 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 26747
I20260619 10:29:35.734341 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.734231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:44269 (local address 127.0.0.1:57632)
0619 10:29:35.734277 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:35.734311 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 10:29:35.734429 26536 catalog_manager.cc:4584] TS e97e2328572445a8a0ea6b285b518795 (127.9.90.65:44269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccb7ff480cc44e99b90b0093f9e62934: Network error: Client connection negotiation failed: client connection to 127.9.90.65:44269: connect: Connection refused (error 111)
I20260619 10:29:35.739502 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 26881
I20260619 10:29:35.743172 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 26519
2026-06-19T10:29:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1831 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T10:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:35Z Disabled control of system clock
I20260619 10:29:35.768077 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45203
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:45203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:29:35.845304 27054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:35.845495 27054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:35.845515 27054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:35.846653 27054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:35.846699 27054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:35.846711 27054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:35.846722 27054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:35.846733 27054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:35.848006 27054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:45203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:35.848217 27054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:35.848467 27054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:35.852766 27059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:35.852734 27062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.852881 27054 server_base.cc:1034] running on GCE node
W20260619 10:29:35.852730 27060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.853173 27054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:35.853456 27054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:35.854583 27054 hybrid_clock.cc:648] HybridClock initialized: now 1781864975854571 us; error 29 us; skew 500 ppm
I20260619 10:29:35.855624 27054 webserver.cc:460] Webserver started at http://127.9.90.126:41819/ using document root <none> and password file <none>
I20260619 10:29:35.855832 27054 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:35.855873 27054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:35.855991 27054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:35.857023 27054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "47f72167c2b74040897c9fd75d8561d9"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "e92978e688de09aba4f238b1a2027e26"
server_key_iv: "6abb7bcffc0c8816fe5278b69df21674"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.857347 27054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "47f72167c2b74040897c9fd75d8561d9"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "e92978e688de09aba4f238b1a2027e26"
server_key_iv: "6abb7bcffc0c8816fe5278b69df21674"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.858479 27054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:29:35.859180 27068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:35.859333 27054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:35.859401 27054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "47f72167c2b74040897c9fd75d8561d9"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "e92978e688de09aba4f238b1a2027e26"
server_key_iv: "6abb7bcffc0c8816fe5278b69df21674"
server_key_version: "encryptionkey@0"
I20260619 10:29:35.859450 27054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:35.883785 27054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:35.884061 27054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:35.884159 27054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:35.887554 27054 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:45203
I20260619 10:29:35.887610 27120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:45203 every 8 connection(s)
I20260619 10:29:35.887899 27054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:35.888773 27121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:35.890938 27121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9: Bootstrap starting.
I20260619 10:29:35.891465 27121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:35.891727 27121 log.cc:824] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:35.891986 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 27054
I20260619 10:29:35.892072 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:35.892273 9577 external_mini_cluster.cc:1428] Setting key c30352cca2f423818ed8129b8828540c
I20260619 10:29:35.892444 27121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9: No bootstrap required, opened a new log
I20260619 10:29:35.894767 27121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f72167c2b74040897c9fd75d8561d9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45203 } }
I20260619 10:29:35.894894 27121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:35.894913 27121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f72167c2b74040897c9fd75d8561d9, State: Initialized, Role: FOLLOWER
I20260619 10:29:35.895022 27121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f72167c2b74040897c9fd75d8561d9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45203 } }
I20260619 10:29:35.895098 27121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:35.895138 27121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:35.895180 27121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:35.895807 27121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f72167c2b74040897c9fd75d8561d9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45203 } }
I20260619 10:29:35.895910 27121 leader_election.cc:302] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47f72167c2b74040897c9fd75d8561d9; no voters:
I20260619 10:29:35.896075 27121 leader_election.cc:288] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:35.896100 27126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:35.896261 27126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [term 1 LEADER]: Becoming Leader. State: Replica: 47f72167c2b74040897c9fd75d8561d9, State: Running, Role: LEADER
I20260619 10:29:35.896318 27121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:35.896417 27126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f72167c2b74040897c9fd75d8561d9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45203 } }
I20260619 10:29:35.896716 27126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47f72167c2b74040897c9fd75d8561d9. Latest consensus state: current_term: 1 leader_uuid: "47f72167c2b74040897c9fd75d8561d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f72167c2b74040897c9fd75d8561d9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45203 } } }
I20260619 10:29:35.897197 27128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47f72167c2b74040897c9fd75d8561d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f72167c2b74040897c9fd75d8561d9" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45203 } } }
I20260619 10:29:35.897363 27128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [sys.catalog]: This master's current role is: LEADER
W20260619 10:29:35.898128 27141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:29:35.898229 27141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:29:35.898284 27140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:35.898294 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:35.892760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.9.90.126:45203)
0619 10:29:35.892957 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:35.892964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:35.892988 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:29:35.893611 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:35.893618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:35.893676 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:35.893824 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:35.894020 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.894878 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.895905 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:35.896459 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:35.896893 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:35.897265 (+ 372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:35.897269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:35.897269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:35.897479 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:35.897524 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:35.897539 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:35.897922 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:35.897954 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 10:29:35.898031 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260619 10:29:35.898437 27140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:35.897279 27126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:35.899771 27140 catalog_manager.cc:1347] Generated new cluster ID: 2322d411dd314e61ab6c57d27b78aa53
I20260619 10:29:35.899821 27140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:35.904134 27140 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:35.904693 27140 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:35.908453 27140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9: Generated new TSK 0
I20260619 10:29:35.908589 27140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:35.910682 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45203
--builtin_ntp_servers=127.9.90.84:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:29:35.987725 27145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:35.987919 27145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:35.987947 27145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:35.989046 27145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:35.989082 27145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:35.989115 27145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:29:35.990453 27145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:35.990670 27145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:35.990928 27145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:35.996451 27151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:35.996488 27153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.996598 27145 server_base.cc:1034] running on GCE node
W20260619 10:29:35.996601 27150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:35.996935 27145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:35.997206 27145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:35.998376 27145 hybrid_clock.cc:648] HybridClock initialized: now 1781864975998358 us; error 34 us; skew 500 ppm
I20260619 10:29:35.999717 27145 webserver.cc:460] Webserver started at http://127.9.90.65:44275/ using document root <none> and password file <none>
I20260619 10:29:35.999974 27145 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:36.000048 27145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:36.000176 27145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:36.001510 27145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "e02af9df25904d0c9ed3959c442d5cc1"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "2f468a32ed87e325d5f12f94fae21efc"
server_key_iv: "d55a27a62ec37b1349a869bfa6562438"
server_key_version: "encryptionkey@0"
I20260619 10:29:36.001942 27145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "e02af9df25904d0c9ed3959c442d5cc1"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "2f468a32ed87e325d5f12f94fae21efc"
server_key_iv: "d55a27a62ec37b1349a869bfa6562438"
server_key_version: "encryptionkey@0"
I20260619 10:29:36.003443 27145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:29:36.004266 27159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:36.004406 27145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:36.004488 27145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "e02af9df25904d0c9ed3959c442d5cc1"
format_stamp: "Formatted at 2026-06-19 10:29:35 on dist-test-slave-hw7q"
server_key: "2f468a32ed87e325d5f12f94fae21efc"
server_key_iv: "d55a27a62ec37b1349a869bfa6562438"
server_key_version: "encryptionkey@0"
I20260619 10:29:36.004549 27145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:36.020280 27145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:36.020520 27145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:36.020614 27145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:36.020798 27145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:36.021065 27145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:36.021098 27145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:36.021121 27145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:36.021135 27145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:36.026377 27145 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:36401
I20260619 10:29:36.026437 27272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:36401 every 8 connection(s)
I20260619 10:29:36.026726 27145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:29:36.031309 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.027369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:58879 (local address 127.9.90.126:45203)
0619 10:29:36.027437 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.027439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.027635 (+ 196us) server_negotiation.cc:407] Connection header received
0619 10:29:36.028088 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.028089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.028101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.028151 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:36.028775 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.029252 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.030229 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.030414 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.030467 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.031120 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:36.031121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:36.031121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:36.031126 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:36.031138 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:36.031141 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:36.031178 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.031232 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.031252 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 10:29:36.031504 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.027356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:45203 (local address 127.9.90.65:58879)
0619 10:29:36.027528 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.027560 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:36.028019 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:36.028230 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:36.028234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:36.028242 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:36.028715 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.028719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.029328 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.029329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.030157 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.030159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.030472 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.030473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.030571 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.031012 (+ 441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:29:36.031012 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:36.031038 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:29:36.031059 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:36.031181 (+ 122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:36.031183 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:36.031186 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:36.031229 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:29:36.031301 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260619 10:29:36.031940 27273 heartbeater.cc:347] Connected to a master server at 127.9.90.126:45203
I20260619 10:29:36.032020 27273 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:36.032191 27273 heartbeater.cc:511] Master 127.9.90.126:45203 requested a full tablet report, sending...
I20260619 10:29:36.032634 27085 ts_manager.cc:194] Registered new tserver with Master: e02af9df25904d0c9ed3959c442d5cc1 (127.9.90.65:36401)
I20260619 10:29:36.033236 27085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.65:58879
I20260619 10:29:36.035331 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 27145
I20260619 10:29:36.035434 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:29:36.035614 9577 external_mini_cluster.cc:1428] Setting key 056ca018c7adc90fffdb05bed0c834d6
I20260619 10:29:36.036417 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45203
--builtin_ntp_servers=127.9.90.84:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:29:36.113176 27276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:36.113358 27276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:36.113377 27276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:36.114487 27276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:36.114524 27276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:36.114548 27276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:29:36.115834 27276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:36.116039 27276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:36.116307 27276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:36.120850 27282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:36.120937 27276 server_base.cc:1034] running on GCE node
W20260619 10:29:36.120885 27284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:36.121081 27281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:36.121302 27276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:36.121486 27276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:36.122608 27276 hybrid_clock.cc:648] HybridClock initialized: now 1781864976122596 us; error 28 us; skew 500 ppm
I20260619 10:29:36.123703 27276 webserver.cc:460] Webserver started at http://127.9.90.66:38115/ using document root <none> and password file <none>
I20260619 10:29:36.123915 27276 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:36.123956 27276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:36.124073 27276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:36.125046 27276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "912adda9ee6a479da4e0947b63002a17"
format_stamp: "Formatted at 2026-06-19 10:29:36 on dist-test-slave-hw7q"
server_key: "e18afcdac7ce5d454bd55837c2422757"
server_key_iv: "0cbd94aa0a8a0429b2b57a927d10c87d"
server_key_version: "encryptionkey@0"
I20260619 10:29:36.125351 27276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "912adda9ee6a479da4e0947b63002a17"
format_stamp: "Formatted at 2026-06-19 10:29:36 on dist-test-slave-hw7q"
server_key: "e18afcdac7ce5d454bd55837c2422757"
server_key_iv: "0cbd94aa0a8a0429b2b57a927d10c87d"
server_key_version: "encryptionkey@0"
I20260619 10:29:36.126498 27276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:29:36.127072 27290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:36.127213 27276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:36.127288 27276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "912adda9ee6a479da4e0947b63002a17"
format_stamp: "Formatted at 2026-06-19 10:29:36 on dist-test-slave-hw7q"
server_key: "e18afcdac7ce5d454bd55837c2422757"
server_key_iv: "0cbd94aa0a8a0429b2b57a927d10c87d"
server_key_version: "encryptionkey@0"
I20260619 10:29:36.127344 27276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:36.167714 27276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:36.168015 27276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:36.168143 27276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:36.168359 27276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:36.168674 27276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:36.168707 27276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:36.168736 27276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:36.168759 27276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:36.174070 27276 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:38133
I20260619 10:29:36.174114 27403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:38133 every 8 connection(s)
I20260619 10:29:36.174425 27276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:29:36.179155 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.175082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:42877 (local address 127.9.90.126:45203)
0619 10:29:36.175158 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.175160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.175369 (+ 209us) server_negotiation.cc:407] Connection header received
0619 10:29:36.175845 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.175846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.175856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.175895 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:36.176534 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.177014 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.177991 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.178174 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.178230 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.178968 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:36.178968 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:36.178968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:36.178972 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:36.178980 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:36.178983 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:36.179013 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.179082 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.179098 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 10:29:36.179365 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.175067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:45203 (local address 127.9.90.66:42877)
0619 10:29:36.175251 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.175286 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:36.175758 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:36.175968 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:36.175972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:36.175980 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:36.176473 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.176477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.177073 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.177074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.177914 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.177916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.178254 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.178254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.178357 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.178847 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:29:36.178848 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:36.178876 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:29:36.178898 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:36.179031 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:36.179034 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:36.179036 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:29:36.179083 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 10:29:36.179159 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260619 10:29:36.179801 27404 heartbeater.cc:347] Connected to a master server at 127.9.90.126:45203
I20260619 10:29:36.179879 27404 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:36.180045 27404 heartbeater.cc:511] Master 127.9.90.126:45203 requested a full tablet report, sending...
I20260619 10:29:36.180380 27085 ts_manager.cc:194] Registered new tserver with Master: 912adda9ee6a479da4e0947b63002a17 (127.9.90.66:38133)
I20260619 10:29:36.180778 27085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.66:42877
I20260619 10:29:36.180831 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 27276
I20260619 10:29:36.180927 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:29:36.181111 9577 external_mini_cluster.cc:1428] Setting key cba0d6f0ede4776f61ff721de8680d7d
I20260619 10:29:36.181917 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45203
--builtin_ntp_servers=127.9.90.84:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:29:36.258982 27407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:36.259181 27407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:36.259209 27407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:36.260306 27407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:36.260342 27407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:36.260373 27407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:29:36.261667 27407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:36.261911 27407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:36.262179 27407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:36.266633 27413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:36.266660 27412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:36.266748 27407 server_base.cc:1034] running on GCE node
W20260619 10:29:36.266669 27415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:36.267048 27407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:36.267261 27407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:36.268392 27407 hybrid_clock.cc:648] HybridClock initialized: now 1781864976268376 us; error 27 us; skew 500 ppm
I20260619 10:29:36.269472 27407 webserver.cc:460] Webserver started at http://127.9.90.67:36377/ using document root <none> and password file <none>
I20260619 10:29:36.269666 27407 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:36.269726 27407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:36.269876 27407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:36.270850 27407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "42b9b893c40040a5ac3bfd4f9fd2f849"
format_stamp: "Formatted at 2026-06-19 10:29:36 on dist-test-slave-hw7q"
server_key: "ef3b51d2311e382cc2fa035d5986cd7b"
server_key_iv: "fef1cd1ceb655116b110f4040bd5b8fd"
server_key_version: "encryptionkey@0"
I20260619 10:29:36.271160 27407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "42b9b893c40040a5ac3bfd4f9fd2f849"
format_stamp: "Formatted at 2026-06-19 10:29:36 on dist-test-slave-hw7q"
server_key: "ef3b51d2311e382cc2fa035d5986cd7b"
server_key_iv: "fef1cd1ceb655116b110f4040bd5b8fd"
server_key_version: "encryptionkey@0"
I20260619 10:29:36.272286 27407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:36.272884 27421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:36.273023 27407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:36.273099 27407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "42b9b893c40040a5ac3bfd4f9fd2f849"
format_stamp: "Formatted at 2026-06-19 10:29:36 on dist-test-slave-hw7q"
server_key: "ef3b51d2311e382cc2fa035d5986cd7b"
server_key_iv: "fef1cd1ceb655116b110f4040bd5b8fd"
server_key_version: "encryptionkey@0"
I20260619 10:29:36.273160 27407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:36.290849 27407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:36.291109 27407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:36.291224 27407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:36.291466 27407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:36.291776 27407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:36.291810 27407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:36.291838 27407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:36.291855 27407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:36.297022 27407 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:37995
I20260619 10:29:36.297076 27534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:37995 every 8 connection(s)
I20260619 10:29:36.297396 27407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:29:36.302055 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.298117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:53327 (local address 127.9.90.126:45203)
0619 10:29:36.298159 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.298161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.298388 (+ 227us) server_negotiation.cc:407] Connection header received
0619 10:29:36.298851 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.298852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.298863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.298910 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:36.299500 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.299986 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.300874 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.301069 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.301114 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.301835 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:36.301837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:36.301837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:36.301842 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:36.301859 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:36.301864 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:36.301914 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.301977 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.301998 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:29:36.302217 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.298137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:45203 (local address 127.9.90.67:53327)
0619 10:29:36.298289 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.298320 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:36.298772 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:36.298961 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:36.298964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:36.298971 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:36.299421 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.299425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.300047 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.300048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.300811 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.300813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.301120 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.301120 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.301221 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.301676 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:29:36.301676 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:36.301703 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:29:36.301734 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:36.301919 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:36.301921 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:36.301923 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:29:36.301969 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:29:36.302041 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260619 10:29:36.302620 27535 heartbeater.cc:347] Connected to a master server at 127.9.90.126:45203
I20260619 10:29:36.302695 27535 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:36.302852 27535 heartbeater.cc:511] Master 127.9.90.126:45203 requested a full tablet report, sending...
I20260619 10:29:36.303184 27085 ts_manager.cc:194] Registered new tserver with Master: 42b9b893c40040a5ac3bfd4f9fd2f849 (127.9.90.67:37995)
I20260619 10:29:36.303525 27085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.67:53327
I20260619 10:29:36.306229 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 27407
I20260619 10:29:36.306327 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:29:36.306495 9577 external_mini_cluster.cc:1428] Setting key c5117bf81b341206e8d0297773ace751
I20260619 10:29:36.307253 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:29:36.311146 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.308489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.9.90.126:45203)
0619 10:29:36.308538 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.308539 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.308605 (+ 66us) server_negotiation.cc:407] Connection header received
0619 10:29:36.308665 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.308666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.308675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.308718 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:36.308859 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.309343 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.310115 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.310308 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.310360 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.310943 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:36.310944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:36.310944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:36.310949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:36.310961 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:36.310965 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:36.311007 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.311051 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.311070 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:29:36.312464 27085 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:29:36.313900 27085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:29:36.319593 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.315685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:37995 (local address 127.0.0.1:45154)
0619 10:29:36.316004 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.316014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:36.316064 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:36.316244 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:36.316247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:36.316250 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:36.316306 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.316308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.317546 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.317547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.318487 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.318489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.319346 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.319347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.319419 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.319434 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 10:29:36.319477 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:29:36.319524 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":30,"thread_start_us":85,"threads_started":1}
I20260619 10:29:36.319672 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.315973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.9.90.65:36401)
0619 10:29:36.316146 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.316150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.316326 (+ 176us) server_negotiation.cc:407] Connection header received
0619 10:29:36.316367 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.316369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.316383 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.316443 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:36.316614 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.317661 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.318889 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.319354 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.319415 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.319503 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.319554 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.319591 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260619 10:29:36.319685 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.315667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:38133 (local address 127.0.0.1:49458)
0619 10:29:36.315789 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.315825 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:36.315880 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:36.316332 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:36.316333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:36.316334 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:36.316363 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.316364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.317397 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.317398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.318652 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.318655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.319502 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.319503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.319553 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.319557 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:36.319594 (+ 37us) client_negotiation.cc:240] Negotiation successful
0619 10:29:36.319627 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":78,"threads_started":1}
I20260619 10:29:36.320003 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.315896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36401 (local address 127.0.0.1:51938)
0619 10:29:36.316215 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.316223 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:36.316269 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:36.316510 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:36.316510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:36.316513 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:36.316552 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.316554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.317731 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.317732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.318824 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.318826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.319425 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.319425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.319482 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.319488 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 10:29:36.319537 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 10:29:36.319577 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":19,"thread_start_us":69,"threads_started":1}
I20260619 10:29:36.320004 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.316037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.9.90.66:38133)
0619 10:29:36.316185 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.316190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.316199 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:29:36.316248 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.316250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.316265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.316336 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:36.316421 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.317322 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.318735 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.319418 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.319775 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.319870 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.319893 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.319929 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260619 10:29:36.320007 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.315744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.9.90.67:37995)
0619 10:29:36.315879 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.315883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.316087 (+ 204us) server_negotiation.cc:407] Connection header received
0619 10:29:36.316130 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.316133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.316147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.316221 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:36.316379 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.317433 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.318567 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.319249 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.319787 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.319868 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.319893 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.319928 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260619 10:29:36.320616 27338 tablet_service.cc:1467] Processing CreateTablet for tablet 9f243c84775e4e25ac47ec51b399140c (DEFAULT_TABLE table=test-table [id=a9e83c4a7992498da36a1a6edcd3702b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:36.320564 27469 tablet_service.cc:1467] Processing CreateTablet for tablet 9f243c84775e4e25ac47ec51b399140c (DEFAULT_TABLE table=test-table [id=a9e83c4a7992498da36a1a6edcd3702b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:36.320922 27338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f243c84775e4e25ac47ec51b399140c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:36.320924 27469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f243c84775e4e25ac47ec51b399140c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:36.323237 27549 tablet_bootstrap.cc:492] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: Bootstrap starting.
I20260619 10:29:36.323426 27550 tablet_bootstrap.cc:492] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17: Bootstrap starting.
I20260619 10:29:36.323596 27207 tablet_service.cc:1467] Processing CreateTablet for tablet 9f243c84775e4e25ac47ec51b399140c (DEFAULT_TABLE table=test-table [id=a9e83c4a7992498da36a1a6edcd3702b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:36.323915 27549 tablet_bootstrap.cc:654] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:36.323931 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f243c84775e4e25ac47ec51b399140c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:36.324024 27550 tablet_bootstrap.cc:654] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:36.324259 27549 log.cc:824] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:36.324295 27550 log.cc:824] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:36.324935 27550 tablet_bootstrap.cc:492] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17: No bootstrap required, opened a new log
I20260619 10:29:36.324992 27550 ts_tablet_manager.cc:1397] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:36.325129 27549 tablet_bootstrap.cc:492] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: No bootstrap required, opened a new log
I20260619 10:29:36.325201 27549 ts_tablet_manager.cc:1397] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:36.327070 27553 tablet_bootstrap.cc:492] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1: Bootstrap starting.
I20260619 10:29:36.326959 27550 raft_consensus.cc:348] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } }
I20260619 10:29:36.327135 27550 raft_consensus.cc:374] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:36.327172 27550 raft_consensus.cc:729] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912adda9ee6a479da4e0947b63002a17, State: Initialized, Role: FOLLOWER
I20260619 10:29:36.327319 27550 consensus_queue.cc:260] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } }
I20260619 10:29:36.327562 27550 ts_tablet_manager.cc:1428] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 10:29:36.327553 27549 raft_consensus.cc:348] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } }
I20260619 10:29:36.327673 27553 tablet_bootstrap.cc:654] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:36.327692 27404 heartbeater.cc:503] Master 127.9.90.126:45203 was elected leader, sending a full tablet report...
I20260619 10:29:36.327701 27549 raft_consensus.cc:374] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:36.327740 27549 raft_consensus.cc:729] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b9b893c40040a5ac3bfd4f9fd2f849, State: Initialized, Role: FOLLOWER
I20260619 10:29:36.327836 27549 consensus_queue.cc:260] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } }
I20260619 10:29:36.327945 27553 log.cc:824] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:36.328052 27549 ts_tablet_manager.cc:1428] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 10:29:36.328352 27535 heartbeater.cc:503] Master 127.9.90.126:45203 was elected leader, sending a full tablet report...
I20260619 10:29:36.328577 27553 tablet_bootstrap.cc:492] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1: No bootstrap required, opened a new log
I20260619 10:29:36.328632 27553 ts_tablet_manager.cc:1397] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:36.330569 27553 raft_consensus.cc:348] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } }
I20260619 10:29:36.330684 27553 raft_consensus.cc:374] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:36.330714 27553 raft_consensus.cc:729] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02af9df25904d0c9ed3959c442d5cc1, State: Initialized, Role: FOLLOWER
I20260619 10:29:36.330817 27553 consensus_queue.cc:260] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } }
I20260619 10:29:36.331039 27553 ts_tablet_manager.cc:1428] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:29:36.331073 27273 heartbeater.cc:503] Master 127.9.90.126:45203 was elected leader, sending a full tablet report...
W20260619 10:29:36.424930 27405 tablet.cc:2369] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:29:36.481942 27555 raft_consensus.cc:482] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:36.482060 27555 raft_consensus.cc:504] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } }
I20260619 10:29:36.482366 27555 leader_election.cc:288] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912adda9ee6a479da4e0947b63002a17 (127.9.90.66:38133), e02af9df25904d0c9ed3959c442d5cc1 (127.9.90.65:36401)
I20260619 10:29:36.485116 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.482625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:51761 (local address 127.9.90.66:38133)
0619 10:29:36.482679 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.482681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.482692 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:29:36.482746 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.482747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.482758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.482805 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:36.482976 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.483511 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.484440 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.484809 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.484859 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.484869 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.485007 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.485044 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 10:29:36.485134 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.482723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:51379 (local address 127.9.90.65:36401)
0619 10:29:36.482770 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.482772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.482888 (+ 116us) server_negotiation.cc:407] Connection header received
0619 10:29:36.482920 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.482921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.482932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.482985 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:36.483111 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.483611 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.484480 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.484828 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.484890 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.484899 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.485033 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.485069 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:29:36.485136 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.482542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:38133 (local address 127.9.90.67:51761)
0619 10:29:36.482602 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.482616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:36.482684 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:36.482827 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:36.482828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:36.482830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:36.482900 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.482902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.483579 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.483580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.484361 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.484363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.484878 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.484879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.484941 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.484944 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:36.484987 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:29:36.485022 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:29:36.485146 27558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.482643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36401 (local address 127.9.90.67:51379)
0619 10:29:36.482828 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.482837 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:36.482882 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:36.482992 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:36.482993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:36.482995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:36.483060 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.483061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.483680 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.483681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.484397 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:36.484399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.484901 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.484902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:36.484968 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.484971 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:36.485022 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:29:36.485055 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":121,"threads_started":1}
I20260619 10:29:36.485436 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f243c84775e4e25ac47ec51b399140c" candidate_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912adda9ee6a479da4e0947b63002a17" is_pre_election: true
I20260619 10:29:36.485579 27358 raft_consensus.cc:2454] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b9b893c40040a5ac3bfd4f9fd2f849 in term 0.
I20260619 10:29:36.485740 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f243c84775e4e25ac47ec51b399140c" candidate_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02af9df25904d0c9ed3959c442d5cc1" is_pre_election: true
I20260619 10:29:36.485780 27422 leader_election.cc:302] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b9b893c40040a5ac3bfd4f9fd2f849, 912adda9ee6a479da4e0947b63002a17; no voters:
I20260619 10:29:36.485915 27227 raft_consensus.cc:2454] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b9b893c40040a5ac3bfd4f9fd2f849 in term 0.
I20260619 10:29:36.485916 27555 raft_consensus.cc:2781] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:29:36.485980 27555 raft_consensus.cc:482] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:29:36.486009 27555 raft_consensus.cc:3037] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:36.486668 27555 raft_consensus.cc:504] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } }
I20260619 10:29:36.486773 27555 leader_election.cc:288] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [CANDIDATE]: Term 1 election: Requested vote from peers 912adda9ee6a479da4e0947b63002a17 (127.9.90.66:38133), e02af9df25904d0c9ed3959c442d5cc1 (127.9.90.65:36401)
I20260619 10:29:36.486923 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f243c84775e4e25ac47ec51b399140c" candidate_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02af9df25904d0c9ed3959c442d5cc1"
I20260619 10:29:36.486943 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f243c84775e4e25ac47ec51b399140c" candidate_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912adda9ee6a479da4e0947b63002a17"
I20260619 10:29:36.486994 27227 raft_consensus.cc:3037] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:36.487015 27358 raft_consensus.cc:3037] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:36.487697 27227 raft_consensus.cc:2454] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b9b893c40040a5ac3bfd4f9fd2f849 in term 1.
I20260619 10:29:36.487699 27358 raft_consensus.cc:2454] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b9b893c40040a5ac3bfd4f9fd2f849 in term 1.
I20260619 10:29:36.487846 27422 leader_election.cc:302] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b9b893c40040a5ac3bfd4f9fd2f849, e02af9df25904d0c9ed3959c442d5cc1; no voters:
I20260619 10:29:36.487941 27555 raft_consensus.cc:2781] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:36.488080 27555 raft_consensus.cc:686] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 1 LEADER]: Becoming Leader. State: Replica: 42b9b893c40040a5ac3bfd4f9fd2f849, State: Running, Role: LEADER
I20260619 10:29:36.488161 27555 consensus_queue.cc:237] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } }
I20260619 10:29:36.488929 27084 catalog_manager.cc:5495] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42b9b893c40040a5ac3bfd4f9fd2f849 (127.9.90.67). New cstate: current_term: 1 leader_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b9b893c40040a5ac3bfd4f9fd2f849" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 37995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 } health_report { overall_health: UNKNOWN } } }
I20260619 10:29:36.500840 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.498759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.9.90.67:37995)
0619 10:29:36.498795 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:36.498797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:36.498806 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:29:36.498885 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:36.498885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:36.498901 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:36.498954 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:29:36.499039 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.499472 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.500160 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:36.500374 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:36.500441 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:36.500529 (+ 88us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:29:36.500623 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:29:36.500683 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:36.500696 (+ 13us) server_negotiation.cc:299] Negotiation successful
0619 10:29:36.500718 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:29:36.502516 27358 raft_consensus.cc:1261] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [term 1 FOLLOWER]: Refusing update from remote peer 42b9b893c40040a5ac3bfd4f9fd2f849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:36.502517 27227 raft_consensus.cc:1261] T 9f243c84775e4e25ac47ec51b399140c P e02af9df25904d0c9ed3959c442d5cc1 [term 1 FOLLOWER]: Refusing update from remote peer 42b9b893c40040a5ac3bfd4f9fd2f849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:36.502810 27555 consensus_queue.cc:1035] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912adda9ee6a479da4e0947b63002a17" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:36.502933 27559 consensus_queue.cc:1035] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e02af9df25904d0c9ed3959c442d5cc1" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 36401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:36.505640 27562 mvcc.cc:205] Tried to move back new op lower bound from 7298518943752007680 to 7298518943696166912. Current Snapshot: MvccSnapshot[applied={T|T < 7298518943752007680 or (T in {7298518943752007680})}]
I20260619 10:29:36.507169 27563 mvcc.cc:205] Tried to move back new op lower bound from 7298518943752007680 to 7298518943696166912. Current Snapshot: MvccSnapshot[applied={T|T < 7298518943752007680 or (T in {7298518943752007680})}]
I20260619 10:29:36.507349 27084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:36.507437 27084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:36.508201 27084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47f72167c2b74040897c9fd75d8561d9: Sending DeleteTablet for 3 replicas of tablet 9f243c84775e4e25ac47ec51b399140c
I20260619 10:29:36.508559 27338 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f243c84775e4e25ac47ec51b399140c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:36 UTC) from {username='slave'} at 127.0.0.1:49458
I20260619 10:29:36.508601 27469 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f243c84775e4e25ac47ec51b399140c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:36 UTC) from {username='slave'} at 127.0.0.1:45154
I20260619 10:29:36.508581 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f243c84775e4e25ac47ec51b399140c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:36 UTC) from {username='slave'} at 127.0.0.1:51938
I20260619 10:29:36.508868 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 27145
I20260619 10:29:36.513877 27572 tablet_replica.cc:330] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17: stopping tablet replica
I20260619 10:29:36.513918 27571 tablet_replica.cc:330] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: stopping tablet replica
I20260619 10:29:36.513983 27572 raft_consensus.cc:2229] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:29:36.514001 27571 raft_consensus.cc:2229] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:29:36.514082 27571 raft_consensus.cc:2258] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:36.514084 27572 raft_consensus.cc:2258] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:36.514500 27571 ts_tablet_manager.cc:1905] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:36.516059 27571 ts_tablet_manager.cc:1918] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:36.516126 27571 log.cc:1196] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/9f243c84775e4e25ac47ec51b399140c
I20260619 10:29:36.516340 27571 ts_tablet_manager.cc:1939] T 9f243c84775e4e25ac47ec51b399140c P 42b9b893c40040a5ac3bfd4f9fd2f849: Deleting consensus metadata
I20260619 10:29:36.516860 27071 catalog_manager.cc:4841] TS 42b9b893c40040a5ac3bfd4f9fd2f849 (127.9.90.67:37995): tablet 9f243c84775e4e25ac47ec51b399140c (table test-table [id=a9e83c4a7992498da36a1a6edcd3702b]) successfully deleted
I20260619 10:29:36.517073 27572 ts_tablet_manager.cc:1905] T 9f243c84775e4e25ac47ec51b399140c P 912adda9ee6a479da4e0947b63002a17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 10:29:36.517591 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:36401 (error 108)
I20260619 10:29:36.517849 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 27276
I20260619 10:29:36.518818 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.518699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:36401 (local address 127.0.0.1:51950)
0619 10:29:36.518748 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.518785 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:36401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 10:29:36.519006 27069 catalog_manager.cc:4584] TS e02af9df25904d0c9ed3959c442d5cc1 (127.9.90.65:36401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f243c84775e4e25ac47ec51b399140c: Network error: Client connection negotiation failed: client connection to 127.9.90.65:36401: connect: Connection refused (error 111)
I20260619 10:29:36.521896 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 27407
I20260619 10:29:36.521971 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:36.521875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:38133 (local address 127.0.0.1:49460)
0619 10:29:36.521916 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:36.521940 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.66:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 10:29:36.526072 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 27054
2026-06-19T10:29:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (783 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:36 dist-test-slave-hw7q krb5kdc[27577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:36 dist-test-slave-hw7q krb5kdc[27577](info): set up 2 sockets
Jun 19 10:29:36 dist-test-slave-hw7q krb5kdc[27577](info): commencing operation
krb5kdc: starting...
W20260619 10:29:37.473351 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:37 dist-test-slave-hw7q krb5kdc[27577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:29:37.582033 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36989
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:36989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:37.659349 27590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:37.659538 27590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:37.659559 27590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:37.660607 27590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:37.660640 27590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:37.660653 27590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:37.660663 27590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:37.660681 27590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:37.662158 27590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:36989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:37.662338 27590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:37.662494 27590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:37.666832 27595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:37.666875 27598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:37.666911 27590 server_base.cc:1034] running on GCE node
W20260619 10:29:37.666875 27596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:37.667289 27590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:37.667541 27590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:37.668674 27590 hybrid_clock.cc:648] HybridClock initialized: now 1781864977668661 us; error 29 us; skew 500 ppm
Jun 19 10:29:37 dist-test-slave-hw7q krb5kdc[27577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864977, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:37.671016 27590 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:29:37.671415 27590 webserver.cc:460] Webserver started at http://127.9.90.126:36761/ using document root <none> and password file <none>
I20260619 10:29:37.671598 27590 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:37.671639 27590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:37.671764 27590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:37.672761 27590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "d9a21b913298458c873d79cb901f08c0"
format_stamp: "Formatted at 2026-06-19 10:29:37 on dist-test-slave-hw7q"
server_key: "891d1c614e60a12c3789fa8783bff392"
server_key_iv: "650fd09f5d1fab9e6c4429ba5638006f"
server_key_version: "encryptionkey@0"
I20260619 10:29:37.673074 27590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "d9a21b913298458c873d79cb901f08c0"
format_stamp: "Formatted at 2026-06-19 10:29:37 on dist-test-slave-hw7q"
server_key: "891d1c614e60a12c3789fa8783bff392"
server_key_iv: "650fd09f5d1fab9e6c4429ba5638006f"
server_key_version: "encryptionkey@0"
I20260619 10:29:37.674429 27590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 10:29:37.675058 27605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:37.675235 27590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:37.675314 27590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "d9a21b913298458c873d79cb901f08c0"
format_stamp: "Formatted at 2026-06-19 10:29:37 on dist-test-slave-hw7q"
server_key: "891d1c614e60a12c3789fa8783bff392"
server_key_iv: "650fd09f5d1fab9e6c4429ba5638006f"
server_key_version: "encryptionkey@0"
I20260619 10:29:37.675365 27590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:37.688560 27590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:37.690467 27590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:37.690575 27590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:37.694092 27590 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:36989
I20260619 10:29:37.694123 27657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:36989 every 8 connection(s)
I20260619 10:29:37.694449 27590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:37.695248 27658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:37.695922 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 27590
I20260619 10:29:37.696009 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:37.696221 9577 external_mini_cluster.cc:1428] Setting key a337364b644a8b061da3d0ada995d9b8
I20260619 10:29:37.697932 27658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0: Bootstrap starting.
I20260619 10:29:37.698459 27658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:37.698716 27658 log.cc:824] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:37.699359 27658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0: No bootstrap required, opened a new log
I20260619 10:29:37.701491 27658 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a21b913298458c873d79cb901f08c0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36989 } }
I20260619 10:29:37.701664 27658 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:37.701719 27658 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a21b913298458c873d79cb901f08c0, State: Initialized, Role: FOLLOWER
I20260619 10:29:37.701869 27658 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a21b913298458c873d79cb901f08c0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36989 } }
I20260619 10:29:37.701946 27658 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:37.701982 27658 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:37.702028 27658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 10:29:37 dist-test-slave-hw7q krb5kdc[27577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:37.702497 27658 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a21b913298458c873d79cb901f08c0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36989 } }
I20260619 10:29:37.702584 27658 leader_election.cc:302] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9a21b913298458c873d79cb901f08c0; no voters:
I20260619 10:29:37.702754 27658 leader_election.cc:288] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:37.702790 27663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:37.702945 27663 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [term 1 LEADER]: Becoming Leader. State: Replica: d9a21b913298458c873d79cb901f08c0, State: Running, Role: LEADER
I20260619 10:29:37.702991 27658 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:37.703634 27663 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a21b913298458c873d79cb901f08c0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36989 } }
I20260619 10:29:37.704392 27664 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a21b913298458c873d79cb901f08c0. Latest consensus state: current_term: 1 leader_uuid: "d9a21b913298458c873d79cb901f08c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a21b913298458c873d79cb901f08c0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36989 } } }
I20260619 10:29:37.704478 27664 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:37.704707 27665 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9a21b913298458c873d79cb901f08c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a21b913298458c873d79cb901f08c0" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36989 } } }
I20260619 10:29:37.704937 27665 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0 [sys.catalog]: This master's current role is: LEADER
W20260619 10:29:37.705202 27679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:29:37.705268 27679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:29:37.705327 27677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:37.705468 27677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:37.706666 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:37.696687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.9.90.126:36989)
0619 10:29:37.696854 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:37.696861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:37.696885 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:29:37.697282 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:37.697288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:37.697332 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:37.697448 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:37.698102 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:37.699034 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:37.700052 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:37.700482 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:37.700566 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:37.703033 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:37.703036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:37.703036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:37.703046 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:37.704932 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:37.705336 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:37.705339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:37.705339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:37.705439 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:37.705657 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:37.705657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:37.705658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:37.705969 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:37.706091 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:37.706337 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 10:29:37.706474 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260619 10:29:37.707188 27677 catalog_manager.cc:1347] Generated new cluster ID: 659d506d32874de4a385fec1e586dfca
I20260619 10:29:37.707249 27677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:37.713872 27677 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:37.714454 27677 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:37.724344 27677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0: Generated new TSK 0
I20260619 10:29:37.724505 27677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:29:37.750416 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36989
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:37.828926 27684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:37.829134 27684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:37.829166 27684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:37.830317 27684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:37.830355 27684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:37.830389 27684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:29:37.831784 27684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:37.831998 27684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:37.832190 27684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:37.836743 27690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:37.836772 27689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:37.836841 27684 server_base.cc:1034] running on GCE node
W20260619 10:29:37.836723 27692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:37.837076 27684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:37.837287 27684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:37.838411 27684 hybrid_clock.cc:648] HybridClock initialized: now 1781864977838404 us; error 28 us; skew 500 ppm
Jun 19 10:29:37 dist-test-slave-hw7q krb5kdc[27577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864977, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:37.840636 27684 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:29:37.841038 27684 webserver.cc:460] Webserver started at http://127.9.90.65:43469/ using document root <none> and password file <none>
I20260619 10:29:37.841228 27684 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:37.841272 27684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:37.841393 27684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:37.842422 27684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "b8cb7daea1a74613961ef10cf0013665"
format_stamp: "Formatted at 2026-06-19 10:29:37 on dist-test-slave-hw7q"
server_key: "0f2f8da37624f7c728a802328b80cc2f"
server_key_iv: "3e343619eec6feb030fabcb2981e6e3c"
server_key_version: "encryptionkey@0"
I20260619 10:29:37.842730 27684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "b8cb7daea1a74613961ef10cf0013665"
format_stamp: "Formatted at 2026-06-19 10:29:37 on dist-test-slave-hw7q"
server_key: "0f2f8da37624f7c728a802328b80cc2f"
server_key_iv: "3e343619eec6feb030fabcb2981e6e3c"
server_key_version: "encryptionkey@0"
I20260619 10:29:37.843863 27684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:37.844488 27699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:37.844655 27684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:37.844736 27684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "b8cb7daea1a74613961ef10cf0013665"
format_stamp: "Formatted at 2026-06-19 10:29:37 on dist-test-slave-hw7q"
server_key: "0f2f8da37624f7c728a802328b80cc2f"
server_key_iv: "3e343619eec6feb030fabcb2981e6e3c"
server_key_version: "encryptionkey@0"
I20260619 10:29:37.844800 27684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:37.869160 27684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:37.870872 27684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:37.870996 27684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:37.871213 27684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:37.871500 27684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:37.871531 27684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:37.871552 27684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:37.871577 27684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:37.876953 27684 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:46229
I20260619 10:29:37.877012 27812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:46229 every 8 connection(s)
I20260619 10:29:37.877318 27684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:29:37 dist-test-slave-hw7q krb5kdc[27577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864977, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:37.885103 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 27684
I20260619 10:29:37.885246 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:29:37.885460 9577 external_mini_cluster.cc:1428] Setting key 2505a7895c0edded02822818a1aae605
I20260619 10:29:37.885928 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:37.878010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:58781 (local address 127.9.90.126:36989)
0619 10:29:37.878050 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:37.878052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:37.878275 (+ 223us) server_negotiation.cc:407] Connection header received
0619 10:29:37.878782 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:37.878783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:37.878795 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:37.878846 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:37.879773 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:37.880272 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:37.881243 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:37.881455 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:37.881509 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:37.883412 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:37.883414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:37.883414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:37.883421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:37.884462 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:37.884866 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:37.884867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:37.884868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:37.884930 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:37.885211 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:37.885213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:37.885213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:37.885353 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:37.885406 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:37.885723 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 10:29:37.885840 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:29:37.886116 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:37.877987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36989 (local address 127.9.90.65:58781)
0619 10:29:37.878151 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:37.878189 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:37.878683 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:37.878938 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:37.878942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:37.879493 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:37.879678 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:37.879682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:37.880353 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:37.880354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:37.881158 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:37.881160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:37.881524 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:37.881525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:37.881638 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:37.882068 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:37.882076 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:37.883338 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:37.884529 (+ 1191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:37.884535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:37.884537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:37.884769 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:37.884995 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:37.884996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:37.884996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:37.885124 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:37.885415 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:37.885421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:37.885580 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 10:29:37.885699 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 10:29:37.885877 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260619 10:29:37.886626 27813 heartbeater.cc:347] Connected to a master server at 127.9.90.126:36989
I20260619 10:29:37.886729 27813 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:37.886901 27813 heartbeater.cc:511] Master 127.9.90.126:36989 requested a full tablet report, sending...
I20260619 10:29:37.887355 27622 ts_manager.cc:194] Registered new tserver with Master: b8cb7daea1a74613961ef10cf0013665 (127.9.90.65:46229)
I20260619 10:29:37.888211 27622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:58781
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:29:37.905694 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36989
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:37.982622 27818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:37.982810 27818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:37.982831 27818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:37.983886 27818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:37.983918 27818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:37.983942 27818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:29:37.985270 27818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:37.985438 27818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:37.985598 27818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:37.990031 27826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:37.990048 27823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:37.990265 27818 server_base.cc:1034] running on GCE node
W20260619 10:29:37.990294 27824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:37.990478 27818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:37.990690 27818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:37.991832 27818 hybrid_clock.cc:648] HybridClock initialized: now 1781864977991821 us; error 27 us; skew 500 ppm
Jun 19 10:29:37 dist-test-slave-hw7q krb5kdc[27577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864977, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:37.994057 27818 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:29:37.994473 27818 webserver.cc:460] Webserver started at http://127.9.90.66:40973/ using document root <none> and password file <none>
I20260619 10:29:37.994654 27818 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:37.994699 27818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:37.994815 27818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:37.995751 27818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "d242842aef1445c39e6ec323f61b0f42"
format_stamp: "Formatted at 2026-06-19 10:29:37 on dist-test-slave-hw7q"
server_key: "5149639061691e2629e6efc5fa4042d4"
server_key_iv: "48aa9e7d04ecf8136b974dc43665b546"
server_key_version: "encryptionkey@0"
I20260619 10:29:37.996057 27818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "d242842aef1445c39e6ec323f61b0f42"
format_stamp: "Formatted at 2026-06-19 10:29:37 on dist-test-slave-hw7q"
server_key: "5149639061691e2629e6efc5fa4042d4"
server_key_iv: "48aa9e7d04ecf8136b974dc43665b546"
server_key_version: "encryptionkey@0"
I20260619 10:29:37.997192 27818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:37.997814 27833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:37.997951 27818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:37.998024 27818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "d242842aef1445c39e6ec323f61b0f42"
format_stamp: "Formatted at 2026-06-19 10:29:37 on dist-test-slave-hw7q"
server_key: "5149639061691e2629e6efc5fa4042d4"
server_key_iv: "48aa9e7d04ecf8136b974dc43665b546"
server_key_version: "encryptionkey@0"
I20260619 10:29:37.998087 27818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:38.007875 27818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:38.009445 27818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:38.009567 27818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:38.009755 27818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:38.010074 27818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:38.010106 27818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.010133 27818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:38.010159 27818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.015451 27818 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:35291
I20260619 10:29:38.015492 27946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:35291 every 8 connection(s)
I20260619 10:29:38.015764 27818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:29:38.019518 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 27818
I20260619 10:29:38.019615 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:29:38.019788 9577 external_mini_cluster.cc:1428] Setting key 7b6349ba4b43340c03ccc5efd06a68fe
Jun 19 10:29:38 dist-test-slave-hw7q krb5kdc[27577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864977, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:38.024417 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.016384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:54141 (local address 127.9.90.126:36989)
0619 10:29:38.016423 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.016425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.016591 (+ 166us) server_negotiation.cc:407] Connection header received
0619 10:29:38.017062 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.017063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.017075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.017126 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:38.018035 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.018516 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.019508 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.019772 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.019831 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.021864 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:38.021865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:38.021866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:38.021873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:38.023114 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:38.023561 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:38.023562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:38.023562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:38.023616 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:38.023853 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:38.023854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:38.023854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:38.023948 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:38.023991 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.024285 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.024354 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:29:38.024648 27949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.016354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36989 (local address 127.9.90.66:54141)
0619 10:29:38.016485 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.016519 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.016971 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.017198 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.017203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.017724 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:38.017949 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.017953 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.018595 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.018596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.019410 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.019413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.019847 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.019848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.019982 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.020458 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:38.020467 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:38.021753 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:38.023200 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:38.023206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:38.023208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:38.023484 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:38.023669 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:38.023670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:38.023670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:38.023798 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:38.023997 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:38.024002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:38.024151 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.024276 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.024434 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260619 10:29:38.025110 27947 heartbeater.cc:347] Connected to a master server at 127.9.90.126:36989
I20260619 10:29:38.025194 27947 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:38.025359 27947 heartbeater.cc:511] Master 127.9.90.126:36989 requested a full tablet report, sending...
I20260619 10:29:38.025709 27622 ts_manager.cc:194] Registered new tserver with Master: d242842aef1445c39e6ec323f61b0f42 (127.9.90.66:35291)
I20260619 10:29:38.026091 27622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:54141
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:29:38.041190 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36989
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:38.117292 27952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:38.117479 27952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:38.117508 27952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:38.118654 27952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:38.118696 27952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:38.118731 27952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:29:38.120085 27952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:36989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:38.120316 27952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:38.120507 27952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:38.124964 27957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:38.124975 27960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:38.125052 27952 server_base.cc:1034] running on GCE node
W20260619 10:29:38.125100 27958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:38.125306 27952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:38.125495 27952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:38.126616 27952 hybrid_clock.cc:648] HybridClock initialized: now 1781864978126602 us; error 29 us; skew 500 ppm
Jun 19 10:29:38 dist-test-slave-hw7q krb5kdc[27577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864978, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:38.128881 27952 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:29:38.129285 27952 webserver.cc:460] Webserver started at http://127.9.90.67:35223/ using document root <none> and password file <none>
I20260619 10:29:38.129482 27952 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:38.129528 27952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:38.129648 27952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:38.130652 27952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "e391c7c31fda4d77867a7232e5d8956a"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "ce3acad82fca133ac79a3128a95f1410"
server_key_iv: "679d51e37776cabeeeb33d62d2fcbab7"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.130975 27952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "e391c7c31fda4d77867a7232e5d8956a"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "ce3acad82fca133ac79a3128a95f1410"
server_key_iv: "679d51e37776cabeeeb33d62d2fcbab7"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.132104 27952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:38.132715 27967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.132853 27952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:38.132928 27952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "e391c7c31fda4d77867a7232e5d8956a"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "ce3acad82fca133ac79a3128a95f1410"
server_key_iv: "679d51e37776cabeeeb33d62d2fcbab7"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.132992 27952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:38.152374 27952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:38.153966 27952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:38.154094 27952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:38.154296 27952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:38.154564 27952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:38.154595 27952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.154623 27952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:38.154640 27952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.159829 27952 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:33589
I20260619 10:29:38.159884 28080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:33589 every 8 connection(s)
I20260619 10:29:38.160141 27952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:29:38.165128 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 27952
I20260619 10:29:38.165232 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:29:38.165405 9577 external_mini_cluster.cc:1428] Setting key e410e0f205e03910edb01b0283753e3a
Jun 19 10:29:38 dist-test-slave-hw7q krb5kdc[27577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864978, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:38.168416 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.160784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:49013 (local address 127.9.90.126:36989)
0619 10:29:38.160834 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.160836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.161029 (+ 193us) server_negotiation.cc:407] Connection header received
0619 10:29:38.161476 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.161477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.161486 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.161537 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:38.162424 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.162981 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.163923 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.164107 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.164153 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.166223 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:38.166225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:38.166225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:38.166232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:38.167246 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:38.167584 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:38.167586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:38.167586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:38.167636 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:38.167845 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:38.167846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:38.167846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:38.167955 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:38.168008 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.168254 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.168330 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:29:38.168566 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.160764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36989 (local address 127.9.90.67:49013)
0619 10:29:38.160912 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.160951 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.161404 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.161588 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.161591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.162159 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:38.162357 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.162362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.163060 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.163061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.163849 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.163851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.164159 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.164159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.164262 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.164645 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:38.164652 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:38.166141 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:38.167314 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:38.167320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:38.167323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:38.167520 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:38.167694 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:38.167695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:38.167695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:38.167793 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:38.168007 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:38.168013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:38.168139 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.168224 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.168367 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260619 10:29:38.168994 28081 heartbeater.cc:347] Connected to a master server at 127.9.90.126:36989
I20260619 10:29:38.169070 28081 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:38.169209 28081 heartbeater.cc:511] Master 127.9.90.126:36989 requested a full tablet report, sending...
I20260619 10:29:38.169569 27622 ts_manager.cc:194] Registered new tserver with Master: e391c7c31fda4d77867a7232e5d8956a (127.9.90.67:33589)
I20260619 10:29:38.169979 27622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:49013
I20260619 10:29:38.176999 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:29:38.183722 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.178326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.9.90.126:36989)
0619 10:29:38.178377 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.178378 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.178490 (+ 112us) server_negotiation.cc:407] Connection header received
0619 10:29:38.178545 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.178546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.178555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.178607 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:38.179125 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.179582 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.180280 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.180454 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.180505 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.181664 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:38.181666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:38.181666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:38.181673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:38.182666 (+ 993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:38.183004 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:38.183004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:38.183005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:38.183054 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:38.183240 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:38.183241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:38.183241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:38.183365 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:38.183455 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.183571 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.183633 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 10:29:38.184976 27622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:29:38.186519 27622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:29:38.192725 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.188703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.9.90.65:46229)
0619 10:29:38.188891 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.188894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.188903 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:29:38.188933 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.188935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.188947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.189005 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.189168 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.190270 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.191738 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.192315 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.192399 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.192471 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.192595 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.192636 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260619 10:29:38.192728 28089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.188386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:46229 (local address 127.0.0.1:35494)
0619 10:29:38.188717 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.188724 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.188765 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.189034 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.189037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.189039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.189091 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.189092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.190352 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.190353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.191655 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.191657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.192409 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.192410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.192487 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.192501 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.192580 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.192625 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":59,"thread_start_us":59,"threads_started":1}
I20260619 10:29:38.192731 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.188551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:33589 (local address 127.0.0.1:34658)
0619 10:29:38.189359 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.189369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.189417 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.189613 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.189615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.189617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.189653 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.189654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.190710 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.190711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.191597 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.191600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.192509 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.192510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.192560 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.192563 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.192603 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.192634 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":72,"threads_started":1}
I20260619 10:29:38.192732 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.188375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:35291 (local address 127.0.0.1:38356)
0619 10:29:38.188490 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.188529 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.188588 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.189222 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.189223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.189225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.189258 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.189260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.190297 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.190298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.191636 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.191639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.192460 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.192462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.192522 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.192526 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.192553 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.192612 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":45,"thread_start_us":50,"threads_started":1}
I20260619 10:29:38.192874 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.188649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.9.90.67:33589)
0619 10:29:38.188822 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.188827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.189477 (+ 650us) server_negotiation.cc:407] Connection header received
0619 10:29:38.189515 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.189517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.189531 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.189596 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.189720 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.190625 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.191666 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.192365 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.192431 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.192483 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.192765 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.192803 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260619 10:29:38.193120 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.188541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.9.90.66:35291)
0619 10:29:38.189103 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.189105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.189112 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:29:38.189140 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.189142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.189154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.189228 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.189332 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.190209 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.191711 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.192365 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.192758 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.192909 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.192935 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.192972 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":85,"threads_started":1}
I20260619 10:29:38.193722 28015 tablet_service.cc:1467] Processing CreateTablet for tablet 45e5a8614e06407dafdab3522b9d180a (DEFAULT_TABLE table=test-table [id=9ec1e232f6154b4ca25c15705e3ed3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:38.193768 27881 tablet_service.cc:1467] Processing CreateTablet for tablet 45e5a8614e06407dafdab3522b9d180a (DEFAULT_TABLE table=test-table [id=9ec1e232f6154b4ca25c15705e3ed3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:38.194067 27881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e5a8614e06407dafdab3522b9d180a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:38.194070 28015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e5a8614e06407dafdab3522b9d180a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:38.194145 27747 tablet_service.cc:1467] Processing CreateTablet for tablet 45e5a8614e06407dafdab3522b9d180a (DEFAULT_TABLE table=test-table [id=9ec1e232f6154b4ca25c15705e3ed3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:38.194402 27747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e5a8614e06407dafdab3522b9d180a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:38.196390 28095 tablet_bootstrap.cc:492] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: Bootstrap starting.
I20260619 10:29:38.196997 28095 tablet_bootstrap.cc:654] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:38.197245 28096 tablet_bootstrap.cc:492] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: Bootstrap starting.
I20260619 10:29:38.197341 28097 tablet_bootstrap.cc:492] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665: Bootstrap starting.
I20260619 10:29:38.197568 28095 log.cc:824] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:38.197909 28097 tablet_bootstrap.cc:654] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:38.197945 28096 tablet_bootstrap.cc:654] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:38.198323 28095 tablet_bootstrap.cc:492] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: No bootstrap required, opened a new log
I20260619 10:29:38.198354 28096 log.cc:824] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:38.198393 28095 ts_tablet_manager.cc:1397] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:38.198444 28097 log.cc:824] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:38.198972 28096 tablet_bootstrap.cc:492] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: No bootstrap required, opened a new log
I20260619 10:29:38.199030 28096 ts_tablet_manager.cc:1397] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:38.199183 28097 tablet_bootstrap.cc:492] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665: No bootstrap required, opened a new log
I20260619 10:29:38.199282 28097 ts_tablet_manager.cc:1397] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 10:29:38.200265 28095 raft_consensus.cc:348] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } }
I20260619 10:29:38.200469 28095 raft_consensus.cc:374] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:38.200534 28095 raft_consensus.cc:729] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d242842aef1445c39e6ec323f61b0f42, State: Initialized, Role: FOLLOWER
I20260619 10:29:38.200563 28096 raft_consensus.cc:348] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } }
I20260619 10:29:38.200666 28095 consensus_queue.cc:260] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } }
I20260619 10:29:38.200740 28096 raft_consensus.cc:374] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:38.200801 28096 raft_consensus.cc:729] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e391c7c31fda4d77867a7232e5d8956a, State: Initialized, Role: FOLLOWER
I20260619 10:29:38.200953 28095 ts_tablet_manager.cc:1428] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:38.200934 28096 consensus_queue.cc:260] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } }
I20260619 10:29:38.200973 27947 heartbeater.cc:503] Master 127.9.90.126:36989 was elected leader, sending a full tablet report...
I20260619 10:29:38.201184 28096 ts_tablet_manager.cc:1428] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:38.201202 28097 raft_consensus.cc:348] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } }
I20260619 10:29:38.201304 28081 heartbeater.cc:503] Master 127.9.90.126:36989 was elected leader, sending a full tablet report...
I20260619 10:29:38.201340 28097 raft_consensus.cc:374] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:38.201429 28097 raft_consensus.cc:729] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8cb7daea1a74613961ef10cf0013665, State: Initialized, Role: FOLLOWER
I20260619 10:29:38.201596 28097 consensus_queue.cc:260] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } }
I20260619 10:29:38.201876 28097 ts_tablet_manager.cc:1428] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 10:29:38.202086 27813 heartbeater.cc:503] Master 127.9.90.126:36989 was elected leader, sending a full tablet report...
W20260619 10:29:38.266258 27948 tablet.cc:2369] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:29:38.356070 28102 raft_consensus.cc:482] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:38.356242 28102 raft_consensus.cc:504] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } }
I20260619 10:29:38.356592 28102 leader_election.cc:288] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d242842aef1445c39e6ec323f61b0f42 (127.9.90.66:35291), b8cb7daea1a74613961ef10cf0013665 (127.9.90.65:46229)
I20260619 10:29:38.359331 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.356827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:45375 (local address 127.9.90.66:35291)
0619 10:29:38.356881 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.356883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.357105 (+ 222us) server_negotiation.cc:407] Connection header received
0619 10:29:38.357148 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.357149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.357161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.357203 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.357330 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.357813 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.358688 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.359037 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.359092 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.359103 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.359227 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.359262 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:29:38.359341 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.356774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:46229 (local address 127.9.90.67:52239)
0619 10:29:38.356839 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.356852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.356907 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.357042 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.357042 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.357044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.357114 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.357115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.357812 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.357813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.358585 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.358587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.359109 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.359110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.359178 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.359187 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.359235 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.359268 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":25}
I20260619 10:29:38.359342 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.356835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:52239 (local address 127.9.90.65:46229)
0619 10:29:38.356881 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.356883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.356894 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:29:38.356976 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.356978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.356990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.357039 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.357189 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.357717 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.358674 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.359039 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.359098 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.359108 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.359249 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.359284 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:29:38.359349 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.356774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:35291 (local address 127.9.90.67:45375)
0619 10:29:38.357047 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.357055 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.357106 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.357215 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.357216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.357217 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.357278 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.357280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.357872 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.357873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.358603 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.358605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.359109 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.359110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.359168 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.359172 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.359214 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.359255 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":129,"thread_start_us":79,"threads_started":1}
I20260619 10:29:38.359658 27766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e5a8614e06407dafdab3522b9d180a" candidate_uuid: "e391c7c31fda4d77867a7232e5d8956a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8cb7daea1a74613961ef10cf0013665" is_pre_election: true
I20260619 10:29:38.359659 27901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e5a8614e06407dafdab3522b9d180a" candidate_uuid: "e391c7c31fda4d77867a7232e5d8956a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d242842aef1445c39e6ec323f61b0f42" is_pre_election: true
I20260619 10:29:38.359824 27766 raft_consensus.cc:2454] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e391c7c31fda4d77867a7232e5d8956a in term 0.
I20260619 10:29:38.359824 27901 raft_consensus.cc:2454] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e391c7c31fda4d77867a7232e5d8956a in term 0.
I20260619 10:29:38.360029 27971 leader_election.cc:302] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d242842aef1445c39e6ec323f61b0f42, e391c7c31fda4d77867a7232e5d8956a; no voters:
I20260619 10:29:38.360148 28102 raft_consensus.cc:2781] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:29:38.360204 28102 raft_consensus.cc:482] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:29:38.360222 28102 raft_consensus.cc:3037] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:38.360886 28102 raft_consensus.cc:504] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } }
I20260619 10:29:38.361016 28102 leader_election.cc:288] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [CANDIDATE]: Term 1 election: Requested vote from peers d242842aef1445c39e6ec323f61b0f42 (127.9.90.66:35291), b8cb7daea1a74613961ef10cf0013665 (127.9.90.65:46229)
I20260619 10:29:38.361155 27901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e5a8614e06407dafdab3522b9d180a" candidate_uuid: "e391c7c31fda4d77867a7232e5d8956a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d242842aef1445c39e6ec323f61b0f42"
I20260619 10:29:38.361161 27766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e5a8614e06407dafdab3522b9d180a" candidate_uuid: "e391c7c31fda4d77867a7232e5d8956a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8cb7daea1a74613961ef10cf0013665"
I20260619 10:29:38.361230 27901 raft_consensus.cc:3037] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:38.361231 27766 raft_consensus.cc:3037] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:38.361943 27766 raft_consensus.cc:2454] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e391c7c31fda4d77867a7232e5d8956a in term 1.
I20260619 10:29:38.361945 27901 raft_consensus.cc:2454] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e391c7c31fda4d77867a7232e5d8956a in term 1.
I20260619 10:29:38.362095 27971 leader_election.cc:302] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d242842aef1445c39e6ec323f61b0f42, e391c7c31fda4d77867a7232e5d8956a; no voters:
I20260619 10:29:38.362164 28102 raft_consensus.cc:2781] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:38.362289 28102 raft_consensus.cc:686] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 1 LEADER]: Becoming Leader. State: Replica: e391c7c31fda4d77867a7232e5d8956a, State: Running, Role: LEADER
I20260619 10:29:38.362371 28102 consensus_queue.cc:237] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } }
I20260619 10:29:38.363185 27622 catalog_manager.cc:5495] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a reported cstate change: term changed from 0 to 1, leader changed from <none> to e391c7c31fda4d77867a7232e5d8956a (127.9.90.67). New cstate: current_term: 1 leader_uuid: "e391c7c31fda4d77867a7232e5d8956a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e391c7c31fda4d77867a7232e5d8956a" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 33589 } health_report { overall_health: HEALTHY } } }
I20260619 10:29:38.374521 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.372467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.9.90.67:33589)
0619 10:29:38.372518 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.372520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.372527 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:29:38.372591 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.372592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.372606 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.372659 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:29:38.372764 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.373197 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.373916 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.374092 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.374139 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.374251 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:29:38.374331 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:29:38.374396 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.374425 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.374432 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:29:38.376060 27766 raft_consensus.cc:1261] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665 [term 1 FOLLOWER]: Refusing update from remote peer e391c7c31fda4d77867a7232e5d8956a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:38.376080 27901 raft_consensus.cc:1261] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [term 1 FOLLOWER]: Refusing update from remote peer e391c7c31fda4d77867a7232e5d8956a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:38.376386 28105 consensus_queue.cc:1035] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d242842aef1445c39e6ec323f61b0f42" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 35291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:38.376497 28102 consensus_queue.cc:1035] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8cb7daea1a74613961ef10cf0013665" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 10:29:38.378003 27814 tablet.cc:2369] T 45e5a8614e06407dafdab3522b9d180a P b8cb7daea1a74613961ef10cf0013665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:29:38.379006 28109 mvcc.cc:205] Tried to move back new op lower bound from 7298518951426134016 to 7298518951372939264. Current Snapshot: MvccSnapshot[applied={T|T < 7298518951426134016}]
I20260619 10:29:38.381197 27622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:38.381294 27622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:38.381960 27622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9a21b913298458c873d79cb901f08c0: Sending DeleteTablet for 3 replicas of tablet 45e5a8614e06407dafdab3522b9d180a
I20260619 10:29:38.382218 27881 tablet_service.cc:1514] Processing DeleteTablet for tablet 45e5a8614e06407dafdab3522b9d180a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:38 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:38356
I20260619 10:29:38.382297 28015 tablet_service.cc:1514] Processing DeleteTablet for tablet 45e5a8614e06407dafdab3522b9d180a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:38 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:34658
I20260619 10:29:38.382504 28118 tablet_replica.cc:330] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: stopping tablet replica
I20260619 10:29:38.382507 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 27684
I20260619 10:29:38.382539 27747 tablet_service.cc:1514] Processing DeleteTablet for tablet 45e5a8614e06407dafdab3522b9d180a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:38 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:35494
I20260619 10:29:38.383189 28118 raft_consensus.cc:2229] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:29:38.383294 28118 raft_consensus.cc:2258] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:38.383826 28119 tablet_replica.cc:330] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: stopping tablet replica
I20260619 10:29:38.383910 28118 ts_tablet_manager.cc:1905] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:38.383956 28119 raft_consensus.cc:2229] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:29:38.384055 28119 raft_consensus.cc:2258] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:38.384965 28119 ts_tablet_manager.cc:1905] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:38.385911 28118 ts_tablet_manager.cc:1918] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:38.386039 28118 log.cc:1196] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/45e5a8614e06407dafdab3522b9d180a
I20260619 10:29:38.386354 28118 ts_tablet_manager.cc:1939] T 45e5a8614e06407dafdab3522b9d180a P d242842aef1445c39e6ec323f61b0f42: Deleting consensus metadata
I20260619 10:29:38.386775 28119 ts_tablet_manager.cc:1918] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:38.386794 27609 catalog_manager.cc:4841] TS d242842aef1445c39e6ec323f61b0f42 (127.9.90.66:35291): tablet 45e5a8614e06407dafdab3522b9d180a (table test-table [id=9ec1e232f6154b4ca25c15705e3ed3d3]) successfully deleted
I20260619 10:29:38.386847 28119 log.cc:1196] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/45e5a8614e06407dafdab3522b9d180a
I20260619 10:29:38.387147 28119 ts_tablet_manager.cc:1939] T 45e5a8614e06407dafdab3522b9d180a P e391c7c31fda4d77867a7232e5d8956a: Deleting consensus metadata
I20260619 10:29:38.387478 27607 catalog_manager.cc:4841] TS e391c7c31fda4d77867a7232e5d8956a (127.9.90.67:33589): tablet 45e5a8614e06407dafdab3522b9d180a (table test-table [id=9ec1e232f6154b4ca25c15705e3ed3d3]) successfully deleted
W20260619 10:29:38.389264 27607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:46229 (error 108)
I20260619 10:29:38.389520 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 27818
I20260619 10:29:38.389670 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.389549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:46229 (local address 127.0.0.1:35498)
0619 10:29:38.389599 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.389638 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 10:29:38.391484 27607 catalog_manager.cc:4584] TS b8cb7daea1a74613961ef10cf0013665 (127.9.90.65:46229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45e5a8614e06407dafdab3522b9d180a: Network error: Client connection negotiation failed: client connection to 127.9.90.65:46229: connect: Connection refused (error 111)
I20260619 10:29:38.393962 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 27952
I20260619 10:29:38.397382 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 27590
2026-06-19T10:29:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1870 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T10:29:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:38Z Disabled control of system clock
I20260619 10:29:38.421590 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45869
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:29:38.499771 28124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:38.499976 28124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:38.499994 28124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:38.501076 28124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:38.501111 28124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:38.501122 28124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:38.501133 28124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:38.501150 28124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:38.502449 28124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:45869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:38.502631 28124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:38.502854 28124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:38.507205 28129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:38.507329 28132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:38.507238 28130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:38.507398 28124 server_base.cc:1034] running on GCE node
I20260619 10:29:38.507669 28124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:38.507925 28124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:38.509074 28124 hybrid_clock.cc:648] HybridClock initialized: now 1781864978509058 us; error 28 us; skew 500 ppm
I20260619 10:29:38.510246 28124 webserver.cc:460] Webserver started at http://127.9.90.126:38389/ using document root <none> and password file <none>
I20260619 10:29:38.510445 28124 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:38.510491 28124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:38.510608 28124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:38.511580 28124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "9625f0e4e0da40a2b1257ad63e219e70"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "d430dfd9ba1a0128f10e06ac7760ed28"
server_key_iv: "ec27f95e820ec549ce4d440de9bd6889"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.511884 28124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "9625f0e4e0da40a2b1257ad63e219e70"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "d430dfd9ba1a0128f10e06ac7760ed28"
server_key_iv: "ec27f95e820ec549ce4d440de9bd6889"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.513051 28124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:38.513661 28138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.513820 28124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:38.513906 28124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "9625f0e4e0da40a2b1257ad63e219e70"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "d430dfd9ba1a0128f10e06ac7760ed28"
server_key_iv: "ec27f95e820ec549ce4d440de9bd6889"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.513967 28124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:38.524501 28124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:38.524760 28124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:38.524878 28124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:38.528302 28124 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:45869
I20260619 10:29:38.528334 28190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:45869 every 8 connection(s)
I20260619 10:29:38.528610 28124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:38.529469 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:38.531646 28191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70: Bootstrap starting.
I20260619 10:29:38.532164 28191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:38.532428 28191 log.cc:824] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:38.533075 28191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70: No bootstrap required, opened a new log
I20260619 10:29:38.534691 28191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9625f0e4e0da40a2b1257ad63e219e70" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45869 } }
I20260619 10:29:38.534822 28191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:38.534843 28191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9625f0e4e0da40a2b1257ad63e219e70, State: Initialized, Role: FOLLOWER
I20260619 10:29:38.534965 28191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9625f0e4e0da40a2b1257ad63e219e70" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45869 } }
I20260619 10:29:38.535037 28191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:38.535068 28191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:38.535115 28191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:38.535151 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 28124
I20260619 10:29:38.535254 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:38.535421 9577 external_mini_cluster.cc:1428] Setting key fe1af5f390302b02db242c865d4ac702
I20260619 10:29:38.535697 28191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9625f0e4e0da40a2b1257ad63e219e70" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45869 } }
I20260619 10:29:38.535801 28191 leader_election.cc:302] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9625f0e4e0da40a2b1257ad63e219e70; no voters:
I20260619 10:29:38.535945 28191 leader_election.cc:288] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:38.536118 28195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:38.536171 28191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:38.536237 28195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [term 1 LEADER]: Becoming Leader. State: Replica: 9625f0e4e0da40a2b1257ad63e219e70, State: Running, Role: LEADER
I20260619 10:29:38.536361 28195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9625f0e4e0da40a2b1257ad63e219e70" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45869 } }
I20260619 10:29:38.536732 28195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9625f0e4e0da40a2b1257ad63e219e70. Latest consensus state: current_term: 1 leader_uuid: "9625f0e4e0da40a2b1257ad63e219e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9625f0e4e0da40a2b1257ad63e219e70" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45869 } } }
I20260619 10:29:38.536803 28195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:38.536868 28198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9625f0e4e0da40a2b1257ad63e219e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9625f0e4e0da40a2b1257ad63e219e70" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45869 } } }
I20260619 10:29:38.536986 28198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70 [sys.catalog]: This master's current role is: LEADER
W20260619 10:29:38.538074 28212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:29:38.538115 28212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:29:38.538197 28203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:38.538352 28203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:38.539829 28203 catalog_manager.cc:1347] Generated new cluster ID: 554ae61a199e435c9b9eeded816d6889
I20260619 10:29:38.539903 28203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:38.540978 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.535852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.9.90.126:45869)
0619 10:29:38.536063 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.536072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.536111 (+ 39us) server_negotiation.cc:407] Connection header received
0619 10:29:38.536220 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.536227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.536294 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.536433 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:38.536656 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.538063 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.539075 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.539602 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.539733 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.540332 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:38.540335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:38.540336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:38.540520 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:38.540556 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:38.540568 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:38.540624 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.540711 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.540789 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260619 10:29:38.554490 28203 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:38.554949 28203 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:38.561369 28203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70: Generated new TSK 0
I20260619 10:29:38.561518 28203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:38.568907 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45869
--builtin_ntp_servers=127.9.90.84:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:29:38.655110 28215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:38.655314 28215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:38.655335 28215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:38.656476 28215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:38.656517 28215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:38.656543 28215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:29:38.657897 28215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:38.658113 28215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:38.658406 28215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:38.663069 28223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:38.663074 28221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:38.663091 28220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:38.663658 28215 server_base.cc:1034] running on GCE node
I20260619 10:29:38.663841 28215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:38.664048 28215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:38.665201 28215 hybrid_clock.cc:648] HybridClock initialized: now 1781864978665188 us; error 34 us; skew 500 ppm
I20260619 10:29:38.666440 28215 webserver.cc:460] Webserver started at http://127.9.90.65:36161/ using document root <none> and password file <none>
I20260619 10:29:38.666652 28215 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:38.666709 28215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:38.666831 28215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:38.667855 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "ba36462362854d88a50ab248d32f7d9b"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "5a5eeb22d7db35eb936de79bf5317813"
server_key_iv: "020ac46d4b959ba05ed738b29d184a85"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.668160 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "ba36462362854d88a50ab248d32f7d9b"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "5a5eeb22d7db35eb936de79bf5317813"
server_key_iv: "020ac46d4b959ba05ed738b29d184a85"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.669361 28215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:38.670131 28229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.670269 28215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:29:38.670346 28215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "ba36462362854d88a50ab248d32f7d9b"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "5a5eeb22d7db35eb936de79bf5317813"
server_key_iv: "020ac46d4b959ba05ed738b29d184a85"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.670403 28215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:38.694744 28215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:38.695045 28215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:38.695173 28215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:38.695389 28215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:38.695704 28215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:38.695739 28215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.695767 28215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:38.695784 28215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.701009 28215 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:44597
I20260619 10:29:38.701066 28342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:44597 every 8 connection(s)
I20260619 10:29:38.701349 28215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:29:38.704087 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 28215
I20260619 10:29:38.704183 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:29:38.704355 9577 external_mini_cluster.cc:1428] Setting key 7074c108fdf11fc1b947cdb1df1b5239
I20260619 10:29:38.705211 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45869
--builtin_ntp_servers=127.9.90.84:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 10:29:38.706640 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.702010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:50359 (local address 127.9.90.126:45869)
0619 10:29:38.702061 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.702063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.702278 (+ 215us) server_negotiation.cc:407] Connection header received
0619 10:29:38.702771 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.702773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.702788 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.702838 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:38.703470 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.704016 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.705272 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.705528 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.705593 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.706388 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:38.706389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:38.706390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:38.706394 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:38.706409 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:38.706413 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:38.706458 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.706532 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.706562 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:29:38.706836 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.701999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:45869 (local address 127.9.90.65:50359)
0619 10:29:38.702162 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.702197 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.702671 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.702906 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.702910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.702921 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:38.703386 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.703391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.704108 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.704109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.705180 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.705182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.705615 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.705616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.705729 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.706260 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:29:38.706260 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:38.706289 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:29:38.706310 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:38.706467 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:38.706472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:38.706476 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.706535 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.706636 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260619 10:29:38.707221 28343 heartbeater.cc:347] Connected to a master server at 127.9.90.126:45869
I20260619 10:29:38.707304 28343 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:38.707472 28343 heartbeater.cc:511] Master 127.9.90.126:45869 requested a full tablet report, sending...
I20260619 10:29:38.707916 28155 ts_manager.cc:194] Registered new tserver with Master: ba36462362854d88a50ab248d32f7d9b (127.9.90.65:44597)
I20260619 10:29:38.708485 28155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.65:50359
W20260619 10:29:38.782467 28346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:38.782650 28346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:38.782670 28346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:38.783782 28346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:38.783820 28346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:38.783845 28346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:29:38.785128 28346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:38.785342 28346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:38.785604 28346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:38.790297 28351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:38.790287 28352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:38.790279 28354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:38.790374 28346 server_base.cc:1034] running on GCE node
I20260619 10:29:38.790676 28346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:38.790910 28346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:38.792048 28346 hybrid_clock.cc:648] HybridClock initialized: now 1781864978792034 us; error 28 us; skew 500 ppm
I20260619 10:29:38.793193 28346 webserver.cc:460] Webserver started at http://127.9.90.66:36883/ using document root <none> and password file <none>
I20260619 10:29:38.793404 28346 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:38.793449 28346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:38.793569 28346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:38.794618 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "7d4adb3db67e4b479faf927ee61bc222"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "218e145621ed220bbd160c1bcbc888ef"
server_key_iv: "111a5cb3c76e9e75c497a5633f14e40b"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.794950 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "7d4adb3db67e4b479faf927ee61bc222"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "218e145621ed220bbd160c1bcbc888ef"
server_key_iv: "111a5cb3c76e9e75c497a5633f14e40b"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.796087 28346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:29:38.796671 28360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.796810 28346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:38.796886 28346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "7d4adb3db67e4b479faf927ee61bc222"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "218e145621ed220bbd160c1bcbc888ef"
server_key_iv: "111a5cb3c76e9e75c497a5633f14e40b"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.796948 28346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:38.818717 28346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:38.818991 28346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:38.819128 28346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:38.819336 28346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:38.819645 28346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:38.819682 28346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.819717 28346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:38.819739 28346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.824990 28346 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:42261
I20260619 10:29:38.825050 28473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:42261 every 8 connection(s)
I20260619 10:29:38.825340 28346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:29:38.829775 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 28346
I20260619 10:29:38.829905 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:29:38.829906 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.826006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:36343 (local address 127.9.90.126:45869)
0619 10:29:38.826061 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.826063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.826261 (+ 198us) server_negotiation.cc:407] Connection header received
0619 10:29:38.826711 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.826711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.826722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.826775 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:38.827378 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.827855 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.828722 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.828894 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.828936 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.829627 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:38.829628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:38.829629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:38.829633 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:38.829646 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:38.829650 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:38.829707 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.829801 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.829828 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:29:38.830089 9577 external_mini_cluster.cc:1428] Setting key 0ba43e7c0bc70821973c2631e1e2a2c5
I20260619 10:29:38.830108 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.825988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:45869 (local address 127.9.90.66:36343)
0619 10:29:38.826152 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.826191 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.826635 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.826828 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.826831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.826839 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:38.827317 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.827321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.827914 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.827914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.828659 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.828661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.828947 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.828947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.829046 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.829514 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:29:38.829515 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:38.829541 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:29:38.829561 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:38.829723 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:38.829726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:38.829728 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.829771 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.829885 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260619 10:29:38.830574 28474 heartbeater.cc:347] Connected to a master server at 127.9.90.126:45869
I20260619 10:29:38.830662 28474 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:38.830843 28474 heartbeater.cc:511] Master 127.9.90.126:45869 requested a full tablet report, sending...
I20260619 10:29:38.831166 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45869
--builtin_ntp_servers=127.9.90.84:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 10:29:38.831172 28154 ts_manager.cc:194] Registered new tserver with Master: 7d4adb3db67e4b479faf927ee61bc222 (127.9.90.66:42261)
I20260619 10:29:38.831552 28154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.66:36343
W20260619 10:29:38.909159 28477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:38.909355 28477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:38.909382 28477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:38.910524 28477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:38.910565 28477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:38.910597 28477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:29:38.911924 28477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:45869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:38.912145 28477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:38.912428 28477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:38.917029 28483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:38.917029 28485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:38.917044 28482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:38.917284 28477 server_base.cc:1034] running on GCE node
I20260619 10:29:38.917486 28477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:38.917691 28477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:38.918834 28477 hybrid_clock.cc:648] HybridClock initialized: now 1781864978918819 us; error 26 us; skew 500 ppm
I20260619 10:29:38.920014 28477 webserver.cc:460] Webserver started at http://127.9.90.67:41655/ using document root <none> and password file <none>
I20260619 10:29:38.920251 28477 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:38.920301 28477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:38.920424 28477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:38.921440 28477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "c0dc78992b504bae860ae9678df41088"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "e3f575610e46b8844f44fa849292dc2a"
server_key_iv: "0f7613899d10f07a0ac19382f42fc6a2"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.921757 28477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "c0dc78992b504bae860ae9678df41088"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "e3f575610e46b8844f44fa849292dc2a"
server_key_iv: "0f7613899d10f07a0ac19382f42fc6a2"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.922916 28477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:29:38.923544 28491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.923682 28477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:38.923754 28477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "c0dc78992b504bae860ae9678df41088"
format_stamp: "Formatted at 2026-06-19 10:29:38 on dist-test-slave-hw7q"
server_key: "e3f575610e46b8844f44fa849292dc2a"
server_key_iv: "0f7613899d10f07a0ac19382f42fc6a2"
server_key_version: "encryptionkey@0"
I20260619 10:29:38.923818 28477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:38.932250 28477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:38.932482 28477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:38.932590 28477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:38.932772 28477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:38.933048 28477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:38.933079 28477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.933105 28477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:38.933130 28477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:38.938395 28477 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:40033
I20260619 10:29:38.938445 28604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:40033 every 8 connection(s)
I20260619 10:29:38.938720 28477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:29:38.943480 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.939287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:59593 (local address 127.9.90.126:45869)
0619 10:29:38.939329 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.939331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.939542 (+ 211us) server_negotiation.cc:407] Connection header received
0619 10:29:38.940048 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.940050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.940068 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.940119 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:38.940758 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.941247 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.942227 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.942413 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.942468 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.943223 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:38.943225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:38.943226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:38.943231 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:38.943246 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:38.943250 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:38.943295 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.943368 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.943400 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:29:38.943701 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.939277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:45869 (local address 127.9.90.67:59593)
0619 10:29:38.939433 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.939464 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.939933 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.940186 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.940190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.940200 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:38.940661 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.940665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.941324 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.941325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.942146 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.942149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.942479 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.942480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.942598 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.943089 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:29:38.943090 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:38.943116 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:29:38.943137 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:38.943301 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:38.943305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:38.943308 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.943367 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.943459 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260619 10:29:38.944103 28605 heartbeater.cc:347] Connected to a master server at 127.9.90.126:45869
I20260619 10:29:38.944188 28605 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:38.944321 28605 heartbeater.cc:511] Master 127.9.90.126:45869 requested a full tablet report, sending...
I20260619 10:29:38.944609 28154 ts_manager.cc:194] Registered new tserver with Master: c0dc78992b504bae860ae9678df41088 (127.9.90.67:40033)
I20260619 10:29:38.944950 28154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.67:59593
I20260619 10:29:38.945166 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 28477
I20260619 10:29:38.945257 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:29:38.945443 9577 external_mini_cluster.cc:1428] Setting key c9df5f4b246c92ae656ed0aeb8b8f600
I20260619 10:29:38.946206 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:29:38.949756 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.947422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.9.90.126:45869)
0619 10:29:38.947463 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.947465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.947558 (+ 93us) server_negotiation.cc:407] Connection header received
0619 10:29:38.947607 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.947607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.947615 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.947656 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:38.947783 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.948215 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.948857 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.949019 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.949057 (+ 38us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.949558 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:38.949559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:38.949560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:38.949563 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:38.949575 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:38.949578 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:38.949637 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.949677 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.949685 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:29:38.951082 28154 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:29:38.952512 28154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:29:38.958200 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.954431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.9.90.65:44597)
0619 10:29:38.954600 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.954605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.954797 (+ 192us) server_negotiation.cc:407] Connection header received
0619 10:29:38.954843 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.954845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.954859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.954919 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.955074 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.956162 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.957118 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.957721 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.957765 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.957872 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.958071 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.958112 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260619 10:29:38.958220 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.954350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:44597 (local address 127.0.0.1:38876)
0619 10:29:38.954609 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.954646 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.954719 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.954930 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.954933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.954934 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.954993 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.954995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.956253 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.956254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.957015 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.957018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.957926 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.957926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.957987 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.957990 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.958043 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.958102 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":77,"threads_started":1}
I20260619 10:29:38.958221 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.954350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:42261 (local address 127.0.0.1:59470)
0619 10:29:38.954673 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.954698 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.954736 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.955048 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.955049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.955050 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.955092 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.955093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.955990 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.955991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.957043 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.957045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.957894 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.957894 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.957969 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.957982 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.958050 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.958116 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":24,"thread_start_us":72,"threads_started":1}
I20260619 10:29:38.958636 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.954545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:40033 (local address 127.0.0.1:44066)
0619 10:29:38.954738 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.954747 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.954797 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.955289 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.955290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.955292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.955326 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.955327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.956476 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.956477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.957489 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.957491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.958435 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.958435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.958492 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.958495 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.958542 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.958576 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":62,"threads_started":1}
I20260619 10:29:38.958773 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.954514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.9.90.66:42261)
0619 10:29:38.954915 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.954919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.954926 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:29:38.954956 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.954957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.954969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.955033 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.955165 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.955916 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.957141 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.957821 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.957880 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.957964 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.958576 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.958620 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":82,"threads_started":1}
I20260619 10:29:38.958839 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.954708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.9.90.67:40033)
0619 10:29:38.955034 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.955039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.955047 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:29:38.955137 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.955139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.955154 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.955232 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.955391 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.956396 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.957563 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.958265 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.958327 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.958403 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.958732 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.958771 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":75,"threads_started":1}
I20260619 10:29:38.958899 28277 tablet_service.cc:1467] Processing CreateTablet for tablet ff502ee1fe834726ad36dd5214a10277 (DEFAULT_TABLE table=test-table [id=d6ae8f7ae719425ba62d392fc65d2d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:38.959244 28277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff502ee1fe834726ad36dd5214a10277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:38.959416 28408 tablet_service.cc:1467] Processing CreateTablet for tablet ff502ee1fe834726ad36dd5214a10277 (DEFAULT_TABLE table=test-table [id=d6ae8f7ae719425ba62d392fc65d2d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:38.959447 28539 tablet_service.cc:1467] Processing CreateTablet for tablet ff502ee1fe834726ad36dd5214a10277 (DEFAULT_TABLE table=test-table [id=d6ae8f7ae719425ba62d392fc65d2d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:38.959720 28408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff502ee1fe834726ad36dd5214a10277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:38.959728 28539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff502ee1fe834726ad36dd5214a10277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:38.962445 28620 tablet_bootstrap.cc:492] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b: Bootstrap starting.
I20260619 10:29:38.962664 28621 tablet_bootstrap.cc:492] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: Bootstrap starting.
I20260619 10:29:38.963019 28620 tablet_bootstrap.cc:654] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:38.963160 28621 tablet_bootstrap.cc:654] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:38.963294 28620 log.cc:824] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:38.963441 28621 log.cc:824] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:38.963742 28619 tablet_bootstrap.cc:492] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: Bootstrap starting.
I20260619 10:29:38.964130 28621 tablet_bootstrap.cc:492] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: No bootstrap required, opened a new log
I20260619 10:29:38.964190 28621 ts_tablet_manager.cc:1397] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:38.964215 28620 tablet_bootstrap.cc:492] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b: No bootstrap required, opened a new log
I20260619 10:29:38.964321 28619 tablet_bootstrap.cc:654] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:38.964356 28620 ts_tablet_manager.cc:1397] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:38.964601 28619 log.cc:824] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:38.965246 28619 tablet_bootstrap.cc:492] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: No bootstrap required, opened a new log
I20260619 10:29:38.965307 28619 ts_tablet_manager.cc:1397] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:38.966147 28621 raft_consensus.cc:348] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } }
I20260619 10:29:38.966280 28621 raft_consensus.cc:374] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:38.966311 28621 raft_consensus.cc:729] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4adb3db67e4b479faf927ee61bc222, State: Initialized, Role: FOLLOWER
I20260619 10:29:38.966415 28621 consensus_queue.cc:260] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } }
I20260619 10:29:38.966646 28621 ts_tablet_manager.cc:1428] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:29:38.966673 28474 heartbeater.cc:503] Master 127.9.90.126:45869 was elected leader, sending a full tablet report...
I20260619 10:29:38.966733 28620 raft_consensus.cc:348] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } }
I20260619 10:29:38.966871 28620 raft_consensus.cc:374] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:38.966909 28620 raft_consensus.cc:729] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba36462362854d88a50ab248d32f7d9b, State: Initialized, Role: FOLLOWER
I20260619 10:29:38.967008 28620 consensus_queue.cc:260] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } }
I20260619 10:29:38.967242 28620 ts_tablet_manager.cc:1428] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:29:38.967288 28343 heartbeater.cc:503] Master 127.9.90.126:45869 was elected leader, sending a full tablet report...
I20260619 10:29:38.967599 28619 raft_consensus.cc:348] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } }
I20260619 10:29:38.967720 28619 raft_consensus.cc:374] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:38.967757 28619 raft_consensus.cc:729] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dc78992b504bae860ae9678df41088, State: Initialized, Role: FOLLOWER
I20260619 10:29:38.967860 28619 consensus_queue.cc:260] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } }
I20260619 10:29:38.968045 28619 ts_tablet_manager.cc:1428] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:29:38.968107 28605 heartbeater.cc:503] Master 127.9.90.126:45869 was elected leader, sending a full tablet report...
I20260619 10:29:38.981120 28627 raft_consensus.cc:482] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:38.981230 28627 raft_consensus.cc:504] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } }
I20260619 10:29:38.981523 28627 leader_election.cc:288] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba36462362854d88a50ab248d32f7d9b (127.9.90.65:44597), 7d4adb3db67e4b479faf927ee61bc222 (127.9.90.66:42261)
I20260619 10:29:38.984087 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.981670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:44597 (local address 127.9.90.67:49315)
0619 10:29:38.981732 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.981744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.981810 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.981948 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.981948 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.981950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.982011 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.982013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.982638 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.982639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.983386 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.983388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.983870 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.983871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.983935 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.983939 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.983983 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.984022 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":2}
I20260619 10:29:38.984138 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.981718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:49315 (local address 127.9.90.65:44597)
0619 10:29:38.981763 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.981765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.981774 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:29:38.981869 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.981871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.981882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.981938 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.982069 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.982570 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.983462 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.983806 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.983866 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.983875 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.984035 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.984073 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 10:29:38.984194 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.981767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:40139 (local address 127.9.90.66:42261)
0619 10:29:38.981827 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.981828 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.981919 (+ 91us) server_negotiation.cc:407] Connection header received
0619 10:29:38.981973 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.981974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.981985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.982036 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:38.982164 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.982589 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.983489 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.983872 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.983935 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.983943 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.984090 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.984129 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 10:29:38.984380 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.981704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:42261 (local address 127.9.90.67:40139)
0619 10:29:38.981861 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:38.981870 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:38.981926 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:38.982041 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:38.982041 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:38.982043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:38.982106 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.982108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.982694 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.982695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.983405 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:38.983407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.983937 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.983938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:38.984007 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.984010 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:38.984279 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 10:29:38.984315 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260619 10:29:38.984397 28297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff502ee1fe834726ad36dd5214a10277" candidate_uuid: "c0dc78992b504bae860ae9678df41088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba36462362854d88a50ab248d32f7d9b" is_pre_election: true
I20260619 10:29:38.984530 28297 raft_consensus.cc:2454] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0dc78992b504bae860ae9678df41088 in term 0.
I20260619 10:29:38.984643 28428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff502ee1fe834726ad36dd5214a10277" candidate_uuid: "c0dc78992b504bae860ae9678df41088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4adb3db67e4b479faf927ee61bc222" is_pre_election: true
I20260619 10:29:38.984774 28428 raft_consensus.cc:2454] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0dc78992b504bae860ae9678df41088 in term 0.
I20260619 10:29:38.984766 28494 leader_election.cc:302] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba36462362854d88a50ab248d32f7d9b, c0dc78992b504bae860ae9678df41088; no voters:
I20260619 10:29:38.984862 28627 raft_consensus.cc:2781] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:29:38.984915 28627 raft_consensus.cc:482] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:29:38.984935 28627 raft_consensus.cc:3037] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:38.985535 28627 raft_consensus.cc:504] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } }
I20260619 10:29:38.985638 28627 leader_election.cc:288] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [CANDIDATE]: Term 1 election: Requested vote from peers ba36462362854d88a50ab248d32f7d9b (127.9.90.65:44597), 7d4adb3db67e4b479faf927ee61bc222 (127.9.90.66:42261)
I20260619 10:29:38.985805 28428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff502ee1fe834726ad36dd5214a10277" candidate_uuid: "c0dc78992b504bae860ae9678df41088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4adb3db67e4b479faf927ee61bc222"
I20260619 10:29:38.985777 28297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff502ee1fe834726ad36dd5214a10277" candidate_uuid: "c0dc78992b504bae860ae9678df41088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba36462362854d88a50ab248d32f7d9b"
I20260619 10:29:38.985870 28428 raft_consensus.cc:3037] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:38.985869 28297 raft_consensus.cc:3037] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:38.986685 28297 raft_consensus.cc:2454] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0dc78992b504bae860ae9678df41088 in term 1.
I20260619 10:29:38.986680 28428 raft_consensus.cc:2454] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0dc78992b504bae860ae9678df41088 in term 1.
I20260619 10:29:38.986896 28494 leader_election.cc:302] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba36462362854d88a50ab248d32f7d9b, c0dc78992b504bae860ae9678df41088; no voters:
I20260619 10:29:38.986977 28627 raft_consensus.cc:2781] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:38.987116 28627 raft_consensus.cc:686] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 1 LEADER]: Becoming Leader. State: Replica: c0dc78992b504bae860ae9678df41088, State: Running, Role: LEADER
I20260619 10:29:38.987226 28627 consensus_queue.cc:237] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } }
I20260619 10:29:38.987922 28155 catalog_manager.cc:5495] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0dc78992b504bae860ae9678df41088 (127.9.90.67). New cstate: current_term: 1 leader_uuid: "c0dc78992b504bae860ae9678df41088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0dc78992b504bae860ae9678df41088" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 40033 } health_report { overall_health: HEALTHY } } }
I20260619 10:29:38.995410 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:38.993268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.9.90.67:40033)
0619 10:29:38.993317 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:38.993319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:38.993326 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:29:38.993391 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:38.993392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:38.993409 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:38.993470 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:29:38.993581 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.994046 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.994742 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:38.994914 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:38.994964 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:38.995103 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:29:38.995193 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:29:38.995271 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:38.995324 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 10:29:38.995331 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:29:38.996819 28428 raft_consensus.cc:1261] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [term 1 FOLLOWER]: Refusing update from remote peer c0dc78992b504bae860ae9678df41088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:38.996851 28297 raft_consensus.cc:1261] T ff502ee1fe834726ad36dd5214a10277 P ba36462362854d88a50ab248d32f7d9b [term 1 FOLLOWER]: Refusing update from remote peer c0dc78992b504bae860ae9678df41088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:38.997130 28627 consensus_queue.cc:1035] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4adb3db67e4b479faf927ee61bc222" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:38.997255 28629 consensus_queue.cc:1035] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba36462362854d88a50ab248d32f7d9b" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:38.999341 28632 mvcc.cc:205] Tried to move back new op lower bound from 7298518953968984064 to 7298518953932292096. Current Snapshot: MvccSnapshot[applied={T|T < 7298518953968984064}]
I20260619 10:29:38.999718 28633 mvcc.cc:205] Tried to move back new op lower bound from 7298518953968984064 to 7298518953932292096. Current Snapshot: MvccSnapshot[applied={T|T < 7298518953968984064 or (T in {7298518953968984064})}]
I20260619 10:29:39.002174 28154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:39.002261 28154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:39.002933 28154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9625f0e4e0da40a2b1257ad63e219e70: Sending DeleteTablet for 3 replicas of tablet ff502ee1fe834726ad36dd5214a10277
I20260619 10:29:39.003225 28277 tablet_service.cc:1514] Processing DeleteTablet for tablet ff502ee1fe834726ad36dd5214a10277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:38 UTC) from {username='slave'} at 127.0.0.1:38876
I20260619 10:29:39.003322 28408 tablet_service.cc:1514] Processing DeleteTablet for tablet ff502ee1fe834726ad36dd5214a10277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:38 UTC) from {username='slave'} at 127.0.0.1:59470
I20260619 10:29:39.003494 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 28215
I20260619 10:29:39.003508 28539 tablet_service.cc:1514] Processing DeleteTablet for tablet ff502ee1fe834726ad36dd5214a10277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:38 UTC) from {username='slave'} at 127.0.0.1:44066
I20260619 10:29:39.005676 28644 tablet_replica.cc:330] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: stopping tablet replica
I20260619 10:29:39.005812 28644 raft_consensus.cc:2229] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:29:39.005934 28644 raft_consensus.cc:2258] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:39.006084 28643 tablet_replica.cc:330] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: stopping tablet replica
I20260619 10:29:39.006175 28643 raft_consensus.cc:2229] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:29:39.006266 28643 raft_consensus.cc:2258] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:39.006616 28644 ts_tablet_manager.cc:1905] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:39.007987 28644 ts_tablet_manager.cc:1918] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:39.008056 28644 log.cc:1196] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/ff502ee1fe834726ad36dd5214a10277
I20260619 10:29:39.008267 28643 ts_tablet_manager.cc:1905] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:39.008296 28644 ts_tablet_manager.cc:1939] T ff502ee1fe834726ad36dd5214a10277 P c0dc78992b504bae860ae9678df41088: Deleting consensus metadata
I20260619 10:29:39.008630 28140 catalog_manager.cc:4841] TS c0dc78992b504bae860ae9678df41088 (127.9.90.67:40033): tablet ff502ee1fe834726ad36dd5214a10277 (table test-table [id=d6ae8f7ae719425ba62d392fc65d2d76]) successfully deleted
I20260619 10:29:39.009821 28643 ts_tablet_manager.cc:1918] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:39.009938 28643 log.cc:1196] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/ff502ee1fe834726ad36dd5214a10277
I20260619 10:29:39.010214 28643 ts_tablet_manager.cc:1939] T ff502ee1fe834726ad36dd5214a10277 P 7d4adb3db67e4b479faf927ee61bc222: Deleting consensus metadata
I20260619 10:29:39.010597 28140 catalog_manager.cc:4841] TS 7d4adb3db67e4b479faf927ee61bc222 (127.9.90.66:42261): tablet ff502ee1fe834726ad36dd5214a10277 (table test-table [id=d6ae8f7ae719425ba62d392fc65d2d76]) successfully deleted
W20260619 10:29:39.011828 28141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:44597 (error 108)
I20260619 10:29:39.012306 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:39.012178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:44597 (local address 127.0.0.1:38882)
0619 10:29:39.012236 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:39.012275 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 10:29:39.012331 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 28346
W20260619 10:29:39.014463 28141 catalog_manager.cc:4584] TS ba36462362854d88a50ab248d32f7d9b (127.9.90.65:44597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff502ee1fe834726ad36dd5214a10277: Network error: Client connection negotiation failed: client connection to 127.9.90.65:44597: connect: Connection refused (error 111)
I20260619 10:29:39.018399 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 28477
I20260619 10:29:39.021932 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 28124
2026-06-19T10:29:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (624 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:39 dist-test-slave-hw7q krb5kdc[28648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:39 dist-test-slave-hw7q krb5kdc[28648](info): set up 2 sockets
Jun 19 10:29:39 dist-test-slave-hw7q krb5kdc[28648](info): commencing operation
krb5kdc: starting...
W20260619 10:29:39.973306 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:40 dist-test-slave-hw7q krb5kdc[28648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:29:40.088269 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:46491
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:42279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:46491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:40.165022 28661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:40.165207 28661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:40.165227 28661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:40.166402 28661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:40.166445 28661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:40.166457 28661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:40.166468 28661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:40.166478 28661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:40.167929 28661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:42279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:46491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:46491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:40.168138 28661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:40.168339 28661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:40.172644 28667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.172672 28666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.172672 28669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:40.172909 28661 server_base.cc:1034] running on GCE node
I20260619 10:29:40.173123 28661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:40.173344 28661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:40.174466 28661 hybrid_clock.cc:648] HybridClock initialized: now 1781864980174459 us; error 29 us; skew 500 ppm
Jun 19 10:29:40 dist-test-slave-hw7q krb5kdc[28648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864980, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:40.176705 28661 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:29:40.177214 28661 webserver.cc:460] Webserver started at http://127.9.90.126:35395/ using document root <none> and password file <none>
I20260619 10:29:40.177403 28661 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:40.177449 28661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:40.177577 28661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:40.178642 28661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "d7f0b487ad7748a2a1ebb90b20a7770b"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "b36c837a6a485ee6fe8ef58ab3724e54"
server_key_iv: "d36ddabb87822c154d43e41837de4933"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.178964 28661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "d7f0b487ad7748a2a1ebb90b20a7770b"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "b36c837a6a485ee6fe8ef58ab3724e54"
server_key_iv: "d36ddabb87822c154d43e41837de4933"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.180229 28661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:29:40.181002 28676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.181195 28661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:29:40.181281 28661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "d7f0b487ad7748a2a1ebb90b20a7770b"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "b36c837a6a485ee6fe8ef58ab3724e54"
server_key_iv: "d36ddabb87822c154d43e41837de4933"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.181347 28661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:40.194499 28661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:40.196218 28661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:40.196339 28661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:40.199956 28661 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:46491
I20260619 10:29:40.199987 28728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:46491 every 8 connection(s)
I20260619 10:29:40.200371 28661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:40.201165 28729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:40.202414 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 28661
I20260619 10:29:40.202512 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:40.202693 9577 external_mini_cluster.cc:1428] Setting key 9946a950406274ccd4a4dfa09958647e
I20260619 10:29:40.203773 28729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b: Bootstrap starting.
I20260619 10:29:40.204422 28729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:40.204689 28729 log.cc:824] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:40.205385 28729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b: No bootstrap required, opened a new log
I20260619 10:29:40.207477 28729 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0b487ad7748a2a1ebb90b20a7770b" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 46491 } }
I20260619 10:29:40.207640 28729 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:40.207674 28729 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f0b487ad7748a2a1ebb90b20a7770b, State: Initialized, Role: FOLLOWER
I20260619 10:29:40.207795 28729 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0b487ad7748a2a1ebb90b20a7770b" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 46491 } }
I20260619 10:29:40.207875 28729 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:40.207912 28729 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:40.207958 28729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:40.208483 28729 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0b487ad7748a2a1ebb90b20a7770b" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 46491 } }
I20260619 10:29:40.208570 28729 leader_election.cc:302] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7f0b487ad7748a2a1ebb90b20a7770b; no voters:
I20260619 10:29:40.208717 28729 leader_election.cc:288] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:40.208750 28734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:40.208860 28734 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [term 1 LEADER]: Becoming Leader. State: Replica: d7f0b487ad7748a2a1ebb90b20a7770b, State: Running, Role: LEADER
I20260619 10:29:40.208940 28734 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0b487ad7748a2a1ebb90b20a7770b" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 46491 } }
I20260619 10:29:40.208969 28729 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:40.209311 28735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7f0b487ad7748a2a1ebb90b20a7770b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0b487ad7748a2a1ebb90b20a7770b" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 46491 } } }
Jun 19 10:29:40 dist-test-slave-hw7q krb5kdc[28648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:40.209393 28735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:40.209524 28734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f0b487ad7748a2a1ebb90b20a7770b. Latest consensus state: current_term: 1 leader_uuid: "d7f0b487ad7748a2a1ebb90b20a7770b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f0b487ad7748a2a1ebb90b20a7770b" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 46491 } } }
I20260619 10:29:40.209579 28734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:40.210211 28741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:40.210657 28741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:40.212401 28741 catalog_manager.cc:1347] Generated new cluster ID: 82a3bd64bc1d4a75bdb8a8538bdd1551
I20260619 10:29:40.212463 28741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:40.213284 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.203160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.9.90.126:46491)
0619 10:29:40.203370 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.203381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.203416 (+ 35us) server_negotiation.cc:407] Connection header received
0619 10:29:40.203939 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.203948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.204013 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.204173 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:40.204727 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.206049 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.206774 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.207407 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.207517 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.209918 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:40.209923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:40.209923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:40.209935 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:40.211935 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.212269 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.212273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.212273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.212359 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.212543 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.212544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.212544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.212764 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:40.212890 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.213004 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.213129 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260619 10:29:40.225732 28741 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:40.226276 28741 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:40.238356 28741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b: Generated new TSK 0
I20260619 10:29:40.238530 28741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
Entry for principal kudu/127.9.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab.
I20260619 10:29:40.260311 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:46491
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:42279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:40.337299 28755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:40.337479 28755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:40.337500 28755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:40.338608 28755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:40.338644 28755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:40.338673 28755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:29:40.340022 28755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:42279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.65.keytab
--principal=kudu/127.9.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:46491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:40.340227 28755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:40.340416 28755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:40.344973 28763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.344990 28761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.344973 28760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:40.345167 28755 server_base.cc:1034] running on GCE node
I20260619 10:29:40.345366 28755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:40.345536 28755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:40.346657 28755 hybrid_clock.cc:648] HybridClock initialized: now 1781864980346648 us; error 27 us; skew 500 ppm
Jun 19 10:29:40 dist-test-slave-hw7q krb5kdc[28648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864980, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:40.348872 28755 init.cc:377] Logged in from keytab as kudu/127.9.90.65@KRBTEST.COM (short username kudu)
I20260619 10:29:40.349288 28755 webserver.cc:460] Webserver started at http://127.9.90.65:36989/ using document root <none> and password file <none>
I20260619 10:29:40.349470 28755 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:40.349514 28755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:40.349632 28755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:40.350620 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "29a4a37438d944019682c296dd8a8d99"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "dd225f645fc1f1508c5218960e283422"
server_key_iv: "ba56d52a0f942d6acb0d59296446fced"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.350929 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "29a4a37438d944019682c296dd8a8d99"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "dd225f645fc1f1508c5218960e283422"
server_key_iv: "ba56d52a0f942d6acb0d59296446fced"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.352207 28755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:40.352881 28770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.353035 28755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:40.353112 28755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "29a4a37438d944019682c296dd8a8d99"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "dd225f645fc1f1508c5218960e283422"
server_key_iv: "ba56d52a0f942d6acb0d59296446fced"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.353181 28755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:40.367897 28755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:40.369884 28755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:40.369997 28755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:40.370201 28755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:40.370486 28755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:40.370517 28755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.370538 28755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:40.370563 28755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.375901 28755 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:45057
I20260619 10:29:40.375937 28883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:45057 every 8 connection(s)
I20260619 10:29:40.376236 28755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
Jun 19 10:29:40 dist-test-slave-hw7q krb5kdc[28648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864980, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.65@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:40.384302 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 28755
I20260619 10:29:40.384400 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:29:40.384483 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.376885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:33341 (local address 127.9.90.126:46491)
0619 10:29:40.376926 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.376928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.377123 (+ 195us) server_negotiation.cc:407] Connection header received
0619 10:29:40.377634 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.377636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.377646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.377703 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:40.378633 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.379132 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.380086 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.380286 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.380328 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.382271 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:40.382272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:40.382273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:40.382278 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:40.383321 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.383664 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.383665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.383665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.383714 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.383902 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.383903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.383903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.384012 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:40.384060 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.384302 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.384391 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:29:40.384609 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.376857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:46491 (local address 127.9.90.65:33341)
0619 10:29:40.377009 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:40.377049 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:40.377526 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:40.377767 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:40.377771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:40.378360 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:40.378549 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.378554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.379244 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.379244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.380026 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.380028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.380340 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.380341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.380435 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.380891 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:40.380899 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:40.382181 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:40.383400 (+ 1219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:40.383405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:40.383407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:40.383602 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:40.383759 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:40.383759 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:40.383759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:40.383847 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:40.384056 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:40.384060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:40.384169 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 10:29:40.384260 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 10:29:40.384406 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260619 10:29:40.384611 9577 external_mini_cluster.cc:1428] Setting key f708754e75ebdb7aa67832bc24021e08
I20260619 10:29:40.385138 28884 heartbeater.cc:347] Connected to a master server at 127.9.90.126:46491
I20260619 10:29:40.385258 28884 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:40.385426 28884 heartbeater.cc:511] Master 127.9.90.126:46491 requested a full tablet report, sending...
I20260619 10:29:40.385942 28693 ts_manager.cc:194] Registered new tserver with Master: 29a4a37438d944019682c296dd8a8d99 (127.9.90.65:45057)
I20260619 10:29:40.386684 28693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.65@KRBTEST.COM'} at 127.9.90.65:33341
WARNING: no policy specified for kudu/127.9.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
Entry for principal kudu/127.9.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab.
I20260619 10:29:40.407756 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:46491
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:42279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:40.490257 28889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:40.490442 28889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:40.490463 28889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:40.491577 28889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:40.491618 28889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:40.491647 28889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:29:40.493064 28889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:42279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.66.keytab
--principal=kudu/127.9.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:46491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:40.493273 28889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:40.493476 28889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:40.498282 28894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.498358 28897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.498378 28895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:40.498415 28889 server_base.cc:1034] running on GCE node
I20260619 10:29:40.498749 28889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:40.498943 28889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:40.500083 28889 hybrid_clock.cc:648] HybridClock initialized: now 1781864980500072 us; error 28 us; skew 500 ppm
Jun 19 10:29:40 dist-test-slave-hw7q krb5kdc[28648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864980, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:40.502391 28889 init.cc:377] Logged in from keytab as kudu/127.9.90.66@KRBTEST.COM (short username kudu)
I20260619 10:29:40.502883 28889 webserver.cc:460] Webserver started at http://127.9.90.66:43271/ using document root <none> and password file <none>
I20260619 10:29:40.503085 28889 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:40.503132 28889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:40.503255 28889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:40.504248 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "60473e30dc0e4a15b935b650fa862e66"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "b242a6f53fb79bc42dcb65f901c1cfb1"
server_key_iv: "ff49dd1d8d910a387b733f229400b3ce"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.504568 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "60473e30dc0e4a15b935b650fa862e66"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "b242a6f53fb79bc42dcb65f901c1cfb1"
server_key_iv: "ff49dd1d8d910a387b733f229400b3ce"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.505748 28889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:29:40.506409 28904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.506556 28889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:40.506632 28889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "60473e30dc0e4a15b935b650fa862e66"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "b242a6f53fb79bc42dcb65f901c1cfb1"
server_key_iv: "ff49dd1d8d910a387b733f229400b3ce"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.506703 28889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:40.520783 28889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:40.522557 28889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:40.522696 28889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:40.522897 28889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:40.523171 28889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:40.523203 28889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.523231 28889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:40.523255 28889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.528671 28889 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:42827
I20260619 10:29:40.528719 29017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:42827 every 8 connection(s)
I20260619 10:29:40.529008 28889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
Jun 19 10:29:40 dist-test-slave-hw7q krb5kdc[28648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864980, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.66@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:40.536922 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.529657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:43841 (local address 127.9.90.126:46491)
0619 10:29:40.529735 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.529738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.529922 (+ 184us) server_negotiation.cc:407] Connection header received
0619 10:29:40.530396 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.530397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.530407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.530456 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:40.531311 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.531794 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.532699 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.532872 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.532915 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.534839 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:40.534840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:40.534840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:40.534847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:40.535905 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.536228 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.536229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.536229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.536273 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.536442 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.536442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.536442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.536532 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:40.536575 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.536802 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.536859 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 10:29:40.537083 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.529632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:46491 (local address 127.9.90.66:43841)
0619 10:29:40.529776 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:40.529842 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:40.530316 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:40.530514 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:40.530518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:40.531052 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:40.531245 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.531249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.531853 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.531853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.532640 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.532642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.532923 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.532923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.533015 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.533458 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:40.533465 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:40.534759 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:40.535967 (+ 1208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:40.535971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:40.535973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:40.536169 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:40.536316 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:40.536316 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:40.536316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:40.536402 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:40.536572 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:40.536575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:40.536708 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 10:29:40.536785 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 10:29:40.536893 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260619 10:29:40.537513 29018 heartbeater.cc:347] Connected to a master server at 127.9.90.126:46491
I20260619 10:29:40.537585 29018 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:40.537716 29018 heartbeater.cc:511] Master 127.9.90.126:46491 requested a full tablet report, sending...
I20260619 10:29:40.538064 28693 ts_manager.cc:194] Registered new tserver with Master: 60473e30dc0e4a15b935b650fa862e66 (127.9.90.66:42827)
I20260619 10:29:40.538420 28693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.66@KRBTEST.COM'} at 127.9.90.66:43841
I20260619 10:29:40.538530 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 28889
I20260619 10:29:40.538620 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:29:40.538815 9577 external_mini_cluster.cc:1428] Setting key 98688cdf159db1ee07e14fd32bebe59b
WARNING: no policy specified for kudu/127.9.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
Entry for principal kudu/127.9.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab.
I20260619 10:29:40.560655 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:46491
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:42279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:40.637460 29023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:40.637645 29023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:40.637666 29023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:40.638800 29023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:40.638840 29023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:40.638865 29023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:29:40.640275 29023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:42279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.67.keytab
--principal=kudu/127.9.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:46491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:40.640451 29023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:40.640618 29023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:40.645282 29031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.645354 29028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.645282 29029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:40.645534 29023 server_base.cc:1034] running on GCE node
I20260619 10:29:40.645697 29023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:40.645938 29023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:40.647073 29023 hybrid_clock.cc:648] HybridClock initialized: now 1781864980647082 us; error 48 us; skew 500 ppm
Jun 19 10:29:40 dist-test-slave-hw7q krb5kdc[28648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864980, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:40.649279 29023 init.cc:377] Logged in from keytab as kudu/127.9.90.67@KRBTEST.COM (short username kudu)
I20260619 10:29:40.649713 29023 webserver.cc:460] Webserver started at http://127.9.90.67:39911/ using document root <none> and password file <none>
I20260619 10:29:40.649950 29023 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:40.649993 29023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:40.650112 29023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:40.651105 29023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "5a72e9e7fd514d89ad28d46c79e7a831"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "799de56195c8729623f173c4009429a9"
server_key_iv: "a2d34ea9938e5620b3238ee9f8a90e83"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.651423 29023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "5a72e9e7fd514d89ad28d46c79e7a831"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "799de56195c8729623f173c4009429a9"
server_key_iv: "a2d34ea9938e5620b3238ee9f8a90e83"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.652575 29023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:29:40.653183 29038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.653311 29023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:40.653388 29023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "5a72e9e7fd514d89ad28d46c79e7a831"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "799de56195c8729623f173c4009429a9"
server_key_iv: "a2d34ea9938e5620b3238ee9f8a90e83"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.653451 29023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:40.671402 29023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:40.673207 29023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:40.673336 29023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:40.673537 29023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:40.673884 29023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:40.673919 29023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.673947 29023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:40.673970 29023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.679399 29023 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:34381
I20260619 10:29:40.679440 29151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:34381 every 8 connection(s)
I20260619 10:29:40.679739 29023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:29:40.684659 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 29023
I20260619 10:29:40.684772 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:29:40.684975 9577 external_mini_cluster.cc:1428] Setting key 53b7cf4bbfe258bc09db59ee2abe0383
Jun 19 10:29:40 dist-test-slave-hw7q krb5kdc[28648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864980, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.67@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:40.688271 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.680412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:33155 (local address 127.9.90.126:46491)
0619 10:29:40.680465 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.680467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.680627 (+ 160us) server_negotiation.cc:407] Connection header received
0619 10:29:40.681115 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.681115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.681125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.681178 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:40.682091 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.682616 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.683660 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.683870 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.683922 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.686189 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:40.686190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:40.686190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:40.686197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:40.687245 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.687602 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.687602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.687602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.687646 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.687824 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.687825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.687825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.687916 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:40.687958 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.688155 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.688210 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:29:40.688480 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.680371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:46491 (local address 127.9.90.67:33155)
0619 10:29:40.680511 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:40.680552 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:40.681043 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:40.681234 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:40.681238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:40.681776 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:40.682012 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.682017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.682749 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.682750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.683586 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.683589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.683943 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.683944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.684052 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.684555 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:29:40.684564 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:40.686106 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:40.687331 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:40.687337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:40.687339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:40.687545 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:40.687695 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:29:40.687696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:29:40.687696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:29:40.687785 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:29:40.687956 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:40.687958 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:40.688068 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 10:29:40.688142 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 10:29:40.688277 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260619 10:29:40.688869 29152 heartbeater.cc:347] Connected to a master server at 127.9.90.126:46491
I20260619 10:29:40.688957 29152 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:40.689123 29152 heartbeater.cc:511] Master 127.9.90.126:46491 requested a full tablet report, sending...
I20260619 10:29:40.689453 28693 ts_manager.cc:194] Registered new tserver with Master: 5a72e9e7fd514d89ad28d46c79e7a831 (127.9.90.67:34381)
I20260619 10:29:40.689836 28693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.90.67@KRBTEST.COM'} at 127.9.90.67:33155
I20260619 10:29:40.696619 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:29:40.703397 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.697930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.9.90.126:46491)
0619 10:29:40.697972 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.697973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.698107 (+ 134us) server_negotiation.cc:407] Connection header received
0619 10:29:40.698148 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.698149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.698158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.698202 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:40.698727 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.699218 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.699881 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.700062 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.700124 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.701221 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:40.701222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:40.701222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:40.701228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:40.702466 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.702773 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.702774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.702775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.702828 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:40.702959 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:40.702960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:40.702960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:40.703084 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:40.703171 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.703269 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.703327 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:29:40.704808 28693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:29:40.706439 28693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:29:40.712415 29161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.708318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:42827 (local address 127.0.0.1:49856)
0619 10:29:40.708588 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:40.708627 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:40.708696 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:40.708964 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:40.708967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:40.708970 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:40.709060 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.709063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.710235 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.710235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.711319 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.711321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.712135 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.712136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.712213 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.712226 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 10:29:40.712289 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 10:29:40.712343 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":48,"threads_started":1}
I20260619 10:29:40.712497 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.708318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:45057 (local address 127.0.0.1:35436)
0619 10:29:40.708751 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:40.708759 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:40.708798 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:40.709242 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:40.709243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:40.709245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:40.709284 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.709286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.710617 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.710619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.711668 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.711670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.712278 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.712279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.712338 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.712341 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:40.712392 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 10:29:40.712429 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":4,"thread_start_us":48,"threads_started":1}
I20260619 10:29:40.712704 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.708408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.9.90.65:45057)
0619 10:29:40.708915 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.708919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.708965 (+ 46us) server_negotiation.cc:407] Connection header received
0619 10:29:40.709005 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.709007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.709106 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.709350 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:40.709422 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.710523 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.711755 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.712205 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.712261 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.712316 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.712407 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.712445 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":81,"threads_started":1}
I20260619 10:29:40.712716 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.708481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.9.90.66:42827)
0619 10:29:40.708642 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.708647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.708677 (+ 30us) server_negotiation.cc:407] Connection header received
0619 10:29:40.708778 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.708780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.708795 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.708874 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:40.709143 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.710124 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.711388 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.711983 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.712047 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.712100 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.712586 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.712629 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260619 10:29:40.712791 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.708486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:34381 (local address 127.0.0.1:58248)
0619 10:29:40.708795 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:40.708803 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:40.708844 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:40.708986 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:40.708987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:40.708989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:40.709041 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.709042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.710443 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.710444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.711596 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.711599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.712602 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.712602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.712658 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.712662 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:40.712698 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 10:29:40.712734 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":22,"thread_start_us":46,"threads_started":1}
I20260619 10:29:40.712992 29165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.708580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.9.90.67:34381)
0619 10:29:40.708735 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.708739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.708851 (+ 112us) server_negotiation.cc:407] Connection header received
0619 10:29:40.708904 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.708907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.708921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.708989 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:40.709170 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.710342 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.711680 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.712349 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.712403 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.712452 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.712880 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.712921 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260619 10:29:40.713323 28952 tablet_service.cc:1467] Processing CreateTablet for tablet 5d42a4b0bc644e08ac106e077833c409 (DEFAULT_TABLE table=test-table [id=eeafec34703a4993871cac3d5ed75be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:40.713323 28818 tablet_service.cc:1467] Processing CreateTablet for tablet 5d42a4b0bc644e08ac106e077833c409 (DEFAULT_TABLE table=test-table [id=eeafec34703a4993871cac3d5ed75be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:40.713660 28952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d42a4b0bc644e08ac106e077833c409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:40.713660 28818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d42a4b0bc644e08ac106e077833c409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:40.716846 29167 tablet_bootstrap.cc:492] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99: Bootstrap starting.
I20260619 10:29:40.717409 29086 tablet_service.cc:1467] Processing CreateTablet for tablet 5d42a4b0bc644e08ac106e077833c409 (DEFAULT_TABLE table=test-table [id=eeafec34703a4993871cac3d5ed75be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:40.717547 29167 tablet_bootstrap.cc:654] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:40.717737 29086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d42a4b0bc644e08ac106e077833c409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:40.717922 29166 tablet_bootstrap.cc:492] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: Bootstrap starting.
I20260619 10:29:40.718111 29167 log.cc:824] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:40.718488 29166 tablet_bootstrap.cc:654] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:40.718858 29167 tablet_bootstrap.cc:492] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99: No bootstrap required, opened a new log
I20260619 10:29:40.718930 29167 ts_tablet_manager.cc:1397] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:40.718940 29166 log.cc:824] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:40.719555 29166 tablet_bootstrap.cc:492] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: No bootstrap required, opened a new log
I20260619 10:29:40.719615 29166 ts_tablet_manager.cc:1397] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:40.720930 29170 tablet_bootstrap.cc:492] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: Bootstrap starting.
I20260619 10:29:40.720841 29167 raft_consensus.cc:348] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } }
I20260619 10:29:40.721007 29167 raft_consensus.cc:374] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:40.721047 29167 raft_consensus.cc:729] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a4a37438d944019682c296dd8a8d99, State: Initialized, Role: FOLLOWER
I20260619 10:29:40.721045 29166 raft_consensus.cc:348] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } }
I20260619 10:29:40.721187 29167 consensus_queue.cc:260] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } }
I20260619 10:29:40.721223 29166 raft_consensus.cc:374] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:40.721261 29166 raft_consensus.cc:729] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60473e30dc0e4a15b935b650fa862e66, State: Initialized, Role: FOLLOWER
I20260619 10:29:40.721381 29166 consensus_queue.cc:260] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } }
I20260619 10:29:40.721428 29167 ts_tablet_manager.cc:1428] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 10:29:40.721530 28884 heartbeater.cc:503] Master 127.9.90.126:46491 was elected leader, sending a full tablet report...
I20260619 10:29:40.721553 29170 tablet_bootstrap.cc:654] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:40.721650 29166 ts_tablet_manager.cc:1428] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:40.721777 29018 heartbeater.cc:503] Master 127.9.90.126:46491 was elected leader, sending a full tablet report...
I20260619 10:29:40.722100 29170 log.cc:824] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:40.722821 29170 tablet_bootstrap.cc:492] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: No bootstrap required, opened a new log
I20260619 10:29:40.722872 29170 ts_tablet_manager.cc:1397] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:40.724278 29170 raft_consensus.cc:348] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } }
I20260619 10:29:40.724403 29170 raft_consensus.cc:374] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:40.724424 29170 raft_consensus.cc:729] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a72e9e7fd514d89ad28d46c79e7a831, State: Initialized, Role: FOLLOWER
I20260619 10:29:40.724529 29170 consensus_queue.cc:260] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } }
I20260619 10:29:40.724761 29170 ts_tablet_manager.cc:1428] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:40.724782 29152 heartbeater.cc:503] Master 127.9.90.126:46491 was elected leader, sending a full tablet report...
I20260619 10:29:40.771004 29174 raft_consensus.cc:482] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:40.771114 29174 raft_consensus.cc:504] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } }
I20260619 10:29:40.771430 29174 leader_election.cc:288] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29a4a37438d944019682c296dd8a8d99 (127.9.90.65:45057), 60473e30dc0e4a15b935b650fa862e66 (127.9.90.66:42827)
I20260619 10:29:40.773975 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.771579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:45057 (local address 127.9.90.67:33263)
0619 10:29:40.771630 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:40.771641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:40.771696 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:40.771820 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:40.771821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:40.771823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:40.771886 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.771888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.772464 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.772465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.773211 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.773213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.773725 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.773726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.773817 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.773821 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:40.773865 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 10:29:40.773902 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":32}
I20260619 10:29:40.773975 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.771672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:33263 (local address 127.9.90.65:45057)
0619 10:29:40.771722 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.771724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.771732 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:29:40.771758 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.771759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.771768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.771810 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:40.771943 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.772400 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.773290 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.773649 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.773705 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.773715 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.773882 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.773917 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 10:29:40.774091 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.771600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:42827 (local address 127.9.90.67:58221)
0619 10:29:40.771752 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:40.771761 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:40.771817 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:40.771933 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:40.771933 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:40.771935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:40.772000 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.772002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.772570 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.772571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.773311 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:40.773313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.773854 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.773855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:40.773928 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.773931 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:40.773976 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:29:40.774013 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260619 10:29:40.774080 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.771653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:58221 (local address 127.9.90.66:42827)
0619 10:29:40.771720 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.771722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.771819 (+ 97us) server_negotiation.cc:407] Connection header received
0619 10:29:40.771869 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.771870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.771881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.771922 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:40.772055 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.772499 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.773390 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.773756 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.773849 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.773859 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.773987 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.774019 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:29:40.774313 28838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d42a4b0bc644e08ac106e077833c409" candidate_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a4a37438d944019682c296dd8a8d99" is_pre_election: true
I20260619 10:29:40.774360 28972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d42a4b0bc644e08ac106e077833c409" candidate_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60473e30dc0e4a15b935b650fa862e66" is_pre_election: true
I20260619 10:29:40.774459 28838 raft_consensus.cc:2454] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a72e9e7fd514d89ad28d46c79e7a831 in term 0.
I20260619 10:29:40.774498 28972 raft_consensus.cc:2454] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a72e9e7fd514d89ad28d46c79e7a831 in term 0.
I20260619 10:29:40.774667 29039 leader_election.cc:302] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a4a37438d944019682c296dd8a8d99, 5a72e9e7fd514d89ad28d46c79e7a831; no voters:
I20260619 10:29:40.774840 29174 raft_consensus.cc:2781] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:29:40.774890 29174 raft_consensus.cc:482] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:29:40.774912 29174 raft_consensus.cc:3037] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:40.775556 29174 raft_consensus.cc:504] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } }
I20260619 10:29:40.775668 29174 leader_election.cc:288] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [CANDIDATE]: Term 1 election: Requested vote from peers 29a4a37438d944019682c296dd8a8d99 (127.9.90.65:45057), 60473e30dc0e4a15b935b650fa862e66 (127.9.90.66:42827)
I20260619 10:29:40.775818 28838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d42a4b0bc644e08ac106e077833c409" candidate_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a4a37438d944019682c296dd8a8d99"
I20260619 10:29:40.775866 28972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d42a4b0bc644e08ac106e077833c409" candidate_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60473e30dc0e4a15b935b650fa862e66"
I20260619 10:29:40.775887 28838 raft_consensus.cc:3037] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:40.775940 28972 raft_consensus.cc:3037] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:40.776549 28838 raft_consensus.cc:2454] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a72e9e7fd514d89ad28d46c79e7a831 in term 1.
I20260619 10:29:40.776595 28972 raft_consensus.cc:2454] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a72e9e7fd514d89ad28d46c79e7a831 in term 1.
I20260619 10:29:40.776708 29039 leader_election.cc:302] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a4a37438d944019682c296dd8a8d99, 5a72e9e7fd514d89ad28d46c79e7a831; no voters:
I20260619 10:29:40.776777 29174 raft_consensus.cc:2781] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:40.776911 29174 raft_consensus.cc:686] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 1 LEADER]: Becoming Leader. State: Replica: 5a72e9e7fd514d89ad28d46c79e7a831, State: Running, Role: LEADER
I20260619 10:29:40.776986 29174 consensus_queue.cc:237] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } }
I20260619 10:29:40.777755 28692 catalog_manager.cc:5495] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a72e9e7fd514d89ad28d46c79e7a831 (127.9.90.67). New cstate: current_term: 1 leader_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a72e9e7fd514d89ad28d46c79e7a831" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 34381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 } health_report { overall_health: UNKNOWN } } }
W20260619 10:29:40.779461 29019 tablet.cc:2369] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:29:40.784032 29165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.781766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.9.90.67:34381)
0619 10:29:40.781825 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.781827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.781873 (+ 46us) server_negotiation.cc:407] Connection header received
0619 10:29:40.781939 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.781940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.781957 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.782018 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:29:40.782132 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.782567 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.783294 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.783530 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.783578 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.783721 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:29:40.783822 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:29:40.783892 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.783927 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.783945 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 10:29:40.785497 28838 raft_consensus.cc:1261] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [term 1 FOLLOWER]: Refusing update from remote peer 5a72e9e7fd514d89ad28d46c79e7a831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:40.785529 28972 raft_consensus.cc:1261] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [term 1 FOLLOWER]: Refusing update from remote peer 5a72e9e7fd514d89ad28d46c79e7a831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:40.785809 29176 consensus_queue.cc:1035] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a4a37438d944019682c296dd8a8d99" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 45057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:40.785923 29174 consensus_queue.cc:1035] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60473e30dc0e4a15b935b650fa862e66" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:40.787969 29179 mvcc.cc:205] Tried to move back new op lower bound from 7298518961295474688 to 7298518961263341568. Current Snapshot: MvccSnapshot[applied={T|T < 7298518961295474688}]
I20260619 10:29:40.788800 29180 mvcc.cc:205] Tried to move back new op lower bound from 7298518961295474688 to 7298518961263341568. Current Snapshot: MvccSnapshot[applied={T|T < 7298518961295474688 or (T in {7298518961295474688})}]
I20260619 10:29:40.790163 28693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:40.790243 28693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:40.790877 28693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7f0b487ad7748a2a1ebb90b20a7770b: Sending DeleteTablet for 3 replicas of tablet 5d42a4b0bc644e08ac106e077833c409
I20260619 10:29:40.791180 29086 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d42a4b0bc644e08ac106e077833c409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:40 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:58248
I20260619 10:29:40.791205 28952 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d42a4b0bc644e08ac106e077833c409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:40 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:49856
I20260619 10:29:40.791205 28818 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d42a4b0bc644e08ac106e077833c409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:40 UTC) from {username='kudu', principal='kudu/127.9.90.126@KRBTEST.COM'} at 127.0.0.1:35436
I20260619 10:29:40.791396 29191 tablet_replica.cc:330] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99: stopping tablet replica
I20260619 10:29:40.791492 29191 raft_consensus.cc:2229] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:29:40.791585 29191 raft_consensus.cc:2258] T 5d42a4b0bc644e08ac106e077833c409 P 29a4a37438d944019682c296dd8a8d99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:40.791708 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 28755
I20260619 10:29:40.792268 29190 tablet_replica.cc:330] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: stopping tablet replica
I20260619 10:29:40.792358 29190 raft_consensus.cc:2229] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:29:40.792443 29190 raft_consensus.cc:2258] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:40.792523 29189 tablet_replica.cc:330] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: stopping tablet replica
I20260619 10:29:40.792595 29189 raft_consensus.cc:2229] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:29:40.792688 29189 raft_consensus.cc:2258] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:40.793113 29190 ts_tablet_manager.cc:1905] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:40.794472 29190 ts_tablet_manager.cc:1918] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:40.794546 29190 log.cc:1196] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/5d42a4b0bc644e08ac106e077833c409
I20260619 10:29:40.794824 29190 ts_tablet_manager.cc:1939] T 5d42a4b0bc644e08ac106e077833c409 P 60473e30dc0e4a15b935b650fa862e66: Deleting consensus metadata
I20260619 10:29:40.795233 28680 catalog_manager.cc:4841] TS 60473e30dc0e4a15b935b650fa862e66 (127.9.90.66:42827): tablet 5d42a4b0bc644e08ac106e077833c409 (table test-table [id=eeafec34703a4993871cac3d5ed75be3]) successfully deleted
W20260619 10:29:40.796847 28677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:45057 (error 108)
I20260619 10:29:40.797286 29161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.797162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:45057 (local address 127.0.0.1:35448)
0619 10:29:40.797212 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:40.797253 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:29:40.797312 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 28889
I20260619 10:29:40.798097 29189 ts_tablet_manager.cc:1905] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 10:29:40.798177 28677 catalog_manager.cc:4584] TS 29a4a37438d944019682c296dd8a8d99 (127.9.90.65:45057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d42a4b0bc644e08ac106e077833c409: Network error: Client connection negotiation failed: client connection to 127.9.90.65:45057: connect: Connection refused (error 111)
I20260619 10:29:40.799453 29189 ts_tablet_manager.cc:1918] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:40.799521 29189 log.cc:1196] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/5d42a4b0bc644e08ac106e077833c409
I20260619 10:29:40.799793 29189 ts_tablet_manager.cc:1939] T 5d42a4b0bc644e08ac106e077833c409 P 5a72e9e7fd514d89ad28d46c79e7a831: Deleting consensus metadata
I20260619 10:29:40.800164 28677 catalog_manager.cc:4841] TS 5a72e9e7fd514d89ad28d46c79e7a831 (127.9.90.67:34381): tablet 5d42a4b0bc644e08ac106e077833c409 (table test-table [id=eeafec34703a4993871cac3d5ed75be3]) successfully deleted
I20260619 10:29:40.801815 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 29023
I20260619 10:29:40.806262 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 28661
2026-06-19T10:29:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1785 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T10:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:40Z Disabled control of system clock
I20260619 10:29:40.830999 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42387
--webserver_interface=127.9.90.126
--webserver_port=0
--builtin_ntp_servers=127.9.90.84:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:29:40.908107 29196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:40.908306 29196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:40.908325 29196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:40.909435 29196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:40.909475 29196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:40.909487 29196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:40.909498 29196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:40.909509 29196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:40.910818 29196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:42387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:40.911041 29196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:40.911305 29196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:40.915659 29201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.915683 29204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:40.915683 29202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:40.915858 29196 server_base.cc:1034] running on GCE node
I20260619 10:29:40.916098 29196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:40.916363 29196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:40.917497 29196 hybrid_clock.cc:648] HybridClock initialized: now 1781864980917484 us; error 29 us; skew 500 ppm
I20260619 10:29:40.918610 29196 webserver.cc:460] Webserver started at http://127.9.90.126:38133/ using document root <none> and password file <none>
I20260619 10:29:40.918835 29196 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:40.918879 29196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:40.919000 29196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:40.920006 29196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "c10143301e3d4ba59d4492f301f86a28"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "f1bba76b4903f25e8ce2b9f98cd257f1"
server_key_iv: "8b1191c1462b62a4ac7b456b3e2c2472"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.920328 29196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "c10143301e3d4ba59d4492f301f86a28"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "f1bba76b4903f25e8ce2b9f98cd257f1"
server_key_iv: "8b1191c1462b62a4ac7b456b3e2c2472"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.921463 29196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:29:40.922127 29210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.922266 29196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:40.922338 29196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "c10143301e3d4ba59d4492f301f86a28"
format_stamp: "Formatted at 2026-06-19 10:29:40 on dist-test-slave-hw7q"
server_key: "f1bba76b4903f25e8ce2b9f98cd257f1"
server_key_iv: "8b1191c1462b62a4ac7b456b3e2c2472"
server_key_version: "encryptionkey@0"
I20260619 10:29:40.922402 29196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:40.934918 29196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:40.935181 29196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:40.935300 29196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:40.938807 29196 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:42387
I20260619 10:29:40.938835 29262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:42387 every 8 connection(s)
I20260619 10:29:40.939123 29196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:40.940023 29263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:40.942267 29263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28: Bootstrap starting.
I20260619 10:29:40.942808 29263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:40.943082 29263 log.cc:824] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:40.943732 29263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28: No bootstrap required, opened a new log
I20260619 10:29:40.944561 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 29196
I20260619 10:29:40.944655 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:40.944823 9577 external_mini_cluster.cc:1428] Setting key db918d416329d874a6c893d3a6f87ddb
I20260619 10:29:40.945429 29263 raft_consensus.cc:348] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10143301e3d4ba59d4492f301f86a28" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42387 } }
I20260619 10:29:40.945580 29263 raft_consensus.cc:374] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:40.945622 29263 raft_consensus.cc:729] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10143301e3d4ba59d4492f301f86a28, State: Initialized, Role: FOLLOWER
I20260619 10:29:40.945746 29263 consensus_queue.cc:260] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10143301e3d4ba59d4492f301f86a28" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42387 } }
I20260619 10:29:40.945845 29263 raft_consensus.cc:388] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:40.945884 29263 raft_consensus.cc:482] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:40.945931 29263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:40.946478 29263 raft_consensus.cc:504] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10143301e3d4ba59d4492f301f86a28" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42387 } }
I20260619 10:29:40.946563 29263 leader_election.cc:302] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c10143301e3d4ba59d4492f301f86a28; no voters:
I20260619 10:29:40.946712 29263 leader_election.cc:288] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:40.946750 29268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:40.946861 29268 raft_consensus.cc:686] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [term 1 LEADER]: Becoming Leader. State: Replica: c10143301e3d4ba59d4492f301f86a28, State: Running, Role: LEADER
I20260619 10:29:40.946959 29263 sys_catalog.cc:564] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:40.946935 29268 consensus_queue.cc:237] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10143301e3d4ba59d4492f301f86a28" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42387 } }
I20260619 10:29:40.947275 29268 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c10143301e3d4ba59d4492f301f86a28. Latest consensus state: current_term: 1 leader_uuid: "c10143301e3d4ba59d4492f301f86a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10143301e3d4ba59d4492f301f86a28" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42387 } } }
I20260619 10:29:40.947355 29268 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:40.947286 29270 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c10143301e3d4ba59d4492f301f86a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10143301e3d4ba59d4492f301f86a28" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 42387 } } }
I20260619 10:29:40.947501 29270 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:40.947541 29274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:40.948006 29274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:40.950629 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:40.945323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.9.90.126:42387)
0619 10:29:40.945538 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:40.945559 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:40.945583 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:29:40.945689 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:40.945696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:40.945761 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:40.945917 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:40.946188 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.947731 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.948630 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:40.949109 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:40.949217 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:40.949927 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:40.949931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:40.949932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:40.950137 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:40.950170 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:40.950181 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:40.950237 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:40.950321 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 10:29:40.950400 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260619 10:29:40.951157 29274 catalog_manager.cc:1347] Generated new cluster ID: 4daad6bf5b5b4306b7fe30933e2350ab
I20260619 10:29:40.951210 29274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:40.979893 29274 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:40.980408 29274 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:40.983502 29274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28: Generated new TSK 0
I20260619 10:29:40.983659 29274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:40.984745 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.65:0
--local_ip_for_outbound_sockets=127.9.90.65
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42387
--builtin_ntp_servers=127.9.90.84:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 10:29:41.078186 29287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:41.078387 29287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:41.078414 29287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:41.079566 29287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:41.079602 29287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:41.079633 29287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.65
I20260619 10:29:41.080938 29287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.65
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.65
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:41.081161 29287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:41.081429 29287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:41.087035 29292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:41.087059 29293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:41.087299 29295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:41.087338 29287 server_base.cc:1034] running on GCE node
I20260619 10:29:41.087670 29287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:41.087893 29287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:41.089031 29287 hybrid_clock.cc:648] HybridClock initialized: now 1781864981089016 us; error 30 us; skew 500 ppm
I20260619 10:29:41.090310 29287 webserver.cc:460] Webserver started at http://127.9.90.65:33275/ using document root <none> and password file <none>
I20260619 10:29:41.090530 29287 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:41.090576 29287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:41.090695 29287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:41.091686 29287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/data/instance:
uuid: "a38a37660eb14437b681f6e0ced19684"
format_stamp: "Formatted at 2026-06-19 10:29:41 on dist-test-slave-hw7q"
server_key: "0bfc0c831f1a8d02368f452053b3d56d"
server_key_iv: "07b4a47aa224beb9d2ded5af6fb3a555"
server_key_version: "encryptionkey@0"
I20260619 10:29:41.091998 29287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance:
uuid: "a38a37660eb14437b681f6e0ced19684"
format_stamp: "Formatted at 2026-06-19 10:29:41 on dist-test-slave-hw7q"
server_key: "0bfc0c831f1a8d02368f452053b3d56d"
server_key_iv: "07b4a47aa224beb9d2ded5af6fb3a555"
server_key_version: "encryptionkey@0"
I20260619 10:29:41.093146 29287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:29:41.093849 29301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.093997 29287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.094063 29287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/wal
uuid: "a38a37660eb14437b681f6e0ced19684"
format_stamp: "Formatted at 2026-06-19 10:29:41 on dist-test-slave-hw7q"
server_key: "0bfc0c831f1a8d02368f452053b3d56d"
server_key_iv: "07b4a47aa224beb9d2ded5af6fb3a555"
server_key_version: "encryptionkey@0"
I20260619 10:29:41.094128 29287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:41.109874 29287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:41.110153 29287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:41.110273 29287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:41.110473 29287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:41.110775 29287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:41.110807 29287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.110828 29287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:41.110839 29287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.116291 29287 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.65:46037
I20260619 10:29:41.116344 29414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.65:46037 every 8 connection(s)
I20260619 10:29:41.116645 29287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/data/info.pb
I20260619 10:29:41.119638 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 29287
I20260619 10:29:41.119730 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-0/wal/instance
I20260619 10:29:41.119899 9577 external_mini_cluster.cc:1428] Setting key 21d626a93530a7281ca56f0a7999ff47
I20260619 10:29:41.120779 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.66:0
--local_ip_for_outbound_sockets=127.9.90.66
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42387
--builtin_ntp_servers=127.9.90.84:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 10:29:41.121826 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.117343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.65:52769 (local address 127.9.90.126:42387)
0619 10:29:41.117414 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.117416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.117630 (+ 214us) server_negotiation.cc:407] Connection header received
0619 10:29:41.118169 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.118170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.118204 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.118244 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:41.118889 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.119380 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.120475 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.120683 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.120755 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.121557 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:41.121558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:41.121558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:41.121563 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:41.121576 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:41.121580 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:41.121623 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.121694 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.121727 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 10:29:41.122020 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.117309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42387 (local address 127.9.90.65:52769)
0619 10:29:41.117505 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:41.117553 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:41.118063 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:41.118300 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:41.118303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:41.118311 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:41.118803 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.118807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.119455 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.119456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.120401 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.120403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.120777 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.120778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.120938 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.121430 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:29:41.121430 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:41.121459 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:29:41.121481 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:41.121624 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:41.121629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:41.121632 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:41.121693 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:29:41.121812 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260619 10:29:41.122545 29415 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42387
I20260619 10:29:41.122637 29415 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:41.122805 29415 heartbeater.cc:511] Master 127.9.90.126:42387 requested a full tablet report, sending...
I20260619 10:29:41.123282 29227 ts_manager.cc:194] Registered new tserver with Master: a38a37660eb14437b681f6e0ced19684 (127.9.90.65:46037)
I20260619 10:29:41.123900 29227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.65:52769
W20260619 10:29:41.197731 29418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:41.197933 29418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:41.197961 29418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:41.199045 29418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:41.199080 29418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:41.199112 29418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.66
I20260619 10:29:41.200392 29418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.66
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.66
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:41.200600 29418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:41.200960 29418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:41.205608 29423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:41.205590 29424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:41.205559 29426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:41.205646 29418 server_base.cc:1034] running on GCE node
I20260619 10:29:41.205981 29418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:41.206184 29418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:41.207335 29418 hybrid_clock.cc:648] HybridClock initialized: now 1781864981207324 us; error 36 us; skew 500 ppm
I20260619 10:29:41.208510 29418 webserver.cc:460] Webserver started at http://127.9.90.66:32941/ using document root <none> and password file <none>
I20260619 10:29:41.208721 29418 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:41.208768 29418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:41.208886 29418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:41.209941 29418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/data/instance:
uuid: "bca38b3f3a164d65bc4b9298f0feda49"
format_stamp: "Formatted at 2026-06-19 10:29:41 on dist-test-slave-hw7q"
server_key: "816e7ac56025810c5a47fc7ecf9209f7"
server_key_iv: "0dc44bcb2d4b2680823625e8a000fbe0"
server_key_version: "encryptionkey@0"
I20260619 10:29:41.210261 29418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance:
uuid: "bca38b3f3a164d65bc4b9298f0feda49"
format_stamp: "Formatted at 2026-06-19 10:29:41 on dist-test-slave-hw7q"
server_key: "816e7ac56025810c5a47fc7ecf9209f7"
server_key_iv: "0dc44bcb2d4b2680823625e8a000fbe0"
server_key_version: "encryptionkey@0"
I20260619 10:29:41.211376 29418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:29:41.211942 29432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.212085 29418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:41.212162 29418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/wal
uuid: "bca38b3f3a164d65bc4b9298f0feda49"
format_stamp: "Formatted at 2026-06-19 10:29:41 on dist-test-slave-hw7q"
server_key: "816e7ac56025810c5a47fc7ecf9209f7"
server_key_iv: "0dc44bcb2d4b2680823625e8a000fbe0"
server_key_version: "encryptionkey@0"
I20260619 10:29:41.212235 29418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:41.228587 29418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:41.228842 29418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:41.228955 29418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:41.229135 29418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:41.229422 29418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:41.229453 29418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.229480 29418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:41.229504 29418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.234637 29418 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.66:46013
I20260619 10:29:41.234680 29545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.66:46013 every 8 connection(s)
I20260619 10:29:41.234957 29418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/data/info.pb
I20260619 10:29:41.235100 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 29418
I20260619 10:29:41.235198 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/wal/instance
I20260619 10:29:41.235395 9577 external_mini_cluster.cc:1428] Setting key ab4450ef4a0fab26706dd654e5b823dd
I20260619 10:29:41.236279 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.67:0
--local_ip_for_outbound_sockets=127.9.90.67
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42387
--builtin_ntp_servers=127.9.90.84:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 10:29:41.240694 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.235726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:52789 (local address 127.9.90.126:42387)
0619 10:29:41.235918 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.235920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.236041 (+ 121us) server_negotiation.cc:407] Connection header received
0619 10:29:41.236542 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.236543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.236554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.236601 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:41.237237 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.237975 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.239356 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.239613 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.239667 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.240477 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:41.240479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:41.240479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:41.240483 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:41.240496 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:41.240500 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:41.240537 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.240611 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.240636 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260619 10:29:41.240942 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.235730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42387 (local address 127.9.90.66:52789)
0619 10:29:41.235919 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:41.235961 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:41.236446 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:41.236667 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:41.236671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:41.236680 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:41.237127 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.237132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.238079 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.238080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.239261 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.239264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.239697 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.239698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.239854 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.240334 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:29:41.240334 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:41.240372 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:29:41.240399 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:41.240542 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:41.240545 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:41.240548 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:41.240613 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 10:29:41.240715 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260619 10:29:41.241436 29546 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42387
I20260619 10:29:41.241513 29546 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:41.241674 29546 heartbeater.cc:511] Master 127.9.90.126:42387 requested a full tablet report, sending...
I20260619 10:29:41.242036 29227 ts_manager.cc:194] Registered new tserver with Master: bca38b3f3a164d65bc4b9298f0feda49 (127.9.90.66:46013)
I20260619 10:29:41.242518 29227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.66:52789
W20260619 10:29:41.313252 29549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:41.313432 29549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:41.313452 29549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:41.314576 29549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:41.314612 29549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:41.314635 29549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.67
I20260619 10:29:41.315943 29549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.67
--webserver_port=0
--tserver_master_addrs=127.9.90.126:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.67
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:41.316123 29549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:41.316346 29549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:41.320849 29554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:41.320921 29549 server_base.cc:1034] running on GCE node
W20260619 10:29:41.320840 29557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:41.320840 29555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:41.321259 29549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:41.321478 29549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:41.322614 29549 hybrid_clock.cc:648] HybridClock initialized: now 1781864981322599 us; error 27 us; skew 500 ppm
I20260619 10:29:41.323766 29549 webserver.cc:460] Webserver started at http://127.9.90.67:42507/ using document root <none> and password file <none>
I20260619 10:29:41.323977 29549 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:41.324025 29549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:41.324144 29549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:41.325183 29549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/data/instance:
uuid: "75c80e028c244ff5ac1c050c39ab0683"
format_stamp: "Formatted at 2026-06-19 10:29:41 on dist-test-slave-hw7q"
server_key: "e889e4c1175c5548cebe449a2e888b49"
server_key_iv: "03b11e3b884460a3b8ace036da4fc7b7"
server_key_version: "encryptionkey@0"
I20260619 10:29:41.325498 29549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance:
uuid: "75c80e028c244ff5ac1c050c39ab0683"
format_stamp: "Formatted at 2026-06-19 10:29:41 on dist-test-slave-hw7q"
server_key: "e889e4c1175c5548cebe449a2e888b49"
server_key_iv: "03b11e3b884460a3b8ace036da4fc7b7"
server_key_version: "encryptionkey@0"
I20260619 10:29:41.326655 29549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:29:41.327253 29563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.327391 29549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:41.327466 29549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/wal
uuid: "75c80e028c244ff5ac1c050c39ab0683"
format_stamp: "Formatted at 2026-06-19 10:29:41 on dist-test-slave-hw7q"
server_key: "e889e4c1175c5548cebe449a2e888b49"
server_key_iv: "03b11e3b884460a3b8ace036da4fc7b7"
server_key_version: "encryptionkey@0"
I20260619 10:29:41.327522 29549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:41.342072 29549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:41.342337 29549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:41.342458 29549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:41.342650 29549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 10:29:41.342931 29549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 10:29:41.342962 29549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.342989 29549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 10:29:41.343007 29549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:41.348457 29549 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.67:44995
I20260619 10:29:41.348527 29676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.67:44995 every 8 connection(s)
I20260619 10:29:41.348806 29549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/data/info.pb
I20260619 10:29:41.350735 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 29549
I20260619 10:29:41.350828 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/wal/instance
I20260619 10:29:41.350987 9577 external_mini_cluster.cc:1428] Setting key c2a3ceeb3d767f62e4946eb004a2a163
I20260619 10:29:41.353895 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.349485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.67:47833 (local address 127.9.90.126:42387)
0619 10:29:41.349563 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.349565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.349765 (+ 200us) server_negotiation.cc:407] Connection header received
0619 10:29:41.350285 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.350286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.350299 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.350347 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:41.351045 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.351702 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.352647 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.352850 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.352903 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.353620 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:41.353621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:41.353622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:41.353626 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:41.353642 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:41.353646 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:41.353691 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.353769 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.353817 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 10:29:41.354087 29679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.349426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:42387 (local address 127.9.90.67:47833)
0619 10:29:41.349631 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:41.349683 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:41.350186 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:41.350428 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:41.350433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:41.350441 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:29:41.350963 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.350968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.351783 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.351784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.352558 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.352560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.352921 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.352922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.353049 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.353500 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 10:29:41.353501 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:29:41.353526 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 10:29:41.353546 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:29:41.353698 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:29:41.353703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:29:41.353706 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:41.353767 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:29:41.353886 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260619 10:29:41.354517 29677 heartbeater.cc:347] Connected to a master server at 127.9.90.126:42387
I20260619 10:29:41.354591 29677 heartbeater.cc:464] Registering TS with master...
I20260619 10:29:41.354753 29677 heartbeater.cc:511] Master 127.9.90.126:42387 requested a full tablet report, sending...
I20260619 10:29:41.355050 29227 ts_manager.cc:194] Registered new tserver with Master: 75c80e028c244ff5ac1c050c39ab0683 (127.9.90.67:44995)
I20260619 10:29:41.355370 29227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.67:47833
I20260619 10:29:41.362528 9577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 10:29:41.366374 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.363775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.9.90.126:42387)
0619 10:29:41.363831 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.363833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.363913 (+ 80us) server_negotiation.cc:407] Connection header received
0619 10:29:41.363968 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.363969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.363981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.364024 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:41.364170 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.364632 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.365334 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.365514 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.365564 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.366153 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:41.366154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:41.366154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:41.366158 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:41.366186 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:41.366190 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:41.366227 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.366276 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.366298 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 10:29:41.367835 29227 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 10:29:41.369305 29227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 10:29:41.375017 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.371232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:44995 (local address 127.0.0.1:59188)
0619 10:29:41.371393 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:41.371431 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:41.371490 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:41.371694 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:41.371697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:41.371699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:41.371757 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.371759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.373003 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.373004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.373936 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.373939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.374744 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.374745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.374816 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.374830 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 10:29:41.374894 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 10:29:41.374940 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":34,"thread_start_us":58,"threads_started":1}
I20260619 10:29:41.375032 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.371393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.9.90.67:44995)
0619 10:29:41.371563 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.371567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.371575 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:29:41.371606 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.371608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.371621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.371693 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:41.371823 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.372914 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.374009 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.374662 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.374735 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.374818 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.374917 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.374954 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260619 10:29:41.375435 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.371236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.66:46013 (local address 127.0.0.1:60666)
0619 10:29:41.371777 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:41.371786 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:41.371829 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:41.372078 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:41.372079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:41.372081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:41.372112 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.372114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.372949 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.372950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.374397 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.374399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.375240 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.375241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.375297 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.375301 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 10:29:41.375338 (+ 37us) client_negotiation.cc:240] Negotiation successful
0619 10:29:41.375371 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":72,"thread_start_us":83,"threads_started":1}
I20260619 10:29:41.375453 29689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.371392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.9.90.66:46013)
0619 10:29:41.371586 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.371591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.371888 (+ 297us) server_negotiation.cc:407] Connection header received
0619 10:29:41.371934 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.371936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.371951 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.372016 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:41.372185 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.372878 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.374459 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.375047 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.375236 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.375323 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.375348 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.375381 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260619 10:29:41.375731 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.371456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.9.90.65:46037)
0619 10:29:41.371610 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.371615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.371693 (+ 78us) server_negotiation.cc:407] Connection header received
0619 10:29:41.371777 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.371780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.371793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.371863 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:41.371955 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.373062 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.374280 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.374976 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.375033 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.375576 (+ 543us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.375605 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.375646 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260619 10:29:41.375682 29611 tablet_service.cc:1467] Processing CreateTablet for tablet ce7269910d2c49b9b390a36f85ef2c32 (DEFAULT_TABLE table=test-table [id=46030e168da24eb79a5d987492ecd511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:41.375844 29685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.371236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:46037 (local address 127.0.0.1:35936)
0619 10:29:41.371639 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:41.371648 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:41.371714 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:41.371866 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:41.371867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:41.371868 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:41.371899 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.371900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.373131 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.373131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.374208 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.374210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.375309 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.375310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.375372 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.375377 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 10:29:41.375420 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:29:41.375459 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260619 10:29:41.376032 29611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7269910d2c49b9b390a36f85ef2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:41.376154 29480 tablet_service.cc:1467] Processing CreateTablet for tablet ce7269910d2c49b9b390a36f85ef2c32 (DEFAULT_TABLE table=test-table [id=46030e168da24eb79a5d987492ecd511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:41.376479 29480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7269910d2c49b9b390a36f85ef2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:41.378564 29691 tablet_bootstrap.cc:492] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: Bootstrap starting.
I20260619 10:29:41.379029 29349 tablet_service.cc:1467] Processing CreateTablet for tablet ce7269910d2c49b9b390a36f85ef2c32 (DEFAULT_TABLE table=test-table [id=46030e168da24eb79a5d987492ecd511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 10:29:41.379261 29691 tablet_bootstrap.cc:654] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:41.379354 29349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7269910d2c49b9b390a36f85ef2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:41.379527 29691 log.cc:824] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:41.379788 29692 tablet_bootstrap.cc:492] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: Bootstrap starting.
I20260619 10:29:41.380218 29691 tablet_bootstrap.cc:492] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: No bootstrap required, opened a new log
I20260619 10:29:41.380340 29691 ts_tablet_manager.cc:1397] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 10:29:41.380354 29692 tablet_bootstrap.cc:654] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:41.380626 29692 log.cc:824] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:41.381342 29692 tablet_bootstrap.cc:492] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: No bootstrap required, opened a new log
I20260619 10:29:41.381404 29692 ts_tablet_manager.cc:1397] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:41.381810 29695 tablet_bootstrap.cc:492] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684: Bootstrap starting.
I20260619 10:29:41.382337 29695 tablet_bootstrap.cc:654] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:41.382715 29695 log.cc:824] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:41.382903 29691 raft_consensus.cc:348] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } }
I20260619 10:29:41.383065 29691 raft_consensus.cc:374] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:41.383103 29691 raft_consensus.cc:729] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c80e028c244ff5ac1c050c39ab0683, State: Initialized, Role: FOLLOWER
I20260619 10:29:41.383210 29691 consensus_queue.cc:260] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } }
I20260619 10:29:41.383373 29695 tablet_bootstrap.cc:492] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684: No bootstrap required, opened a new log
I20260619 10:29:41.383436 29695 ts_tablet_manager.cc:1397] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 10:29:41.383448 29691 ts_tablet_manager.cc:1428] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:29:41.383551 29677 heartbeater.cc:503] Master 127.9.90.126:42387 was elected leader, sending a full tablet report...
I20260619 10:29:41.383944 29692 raft_consensus.cc:348] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } }
I20260619 10:29:41.384054 29692 raft_consensus.cc:374] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:41.384083 29692 raft_consensus.cc:729] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca38b3f3a164d65bc4b9298f0feda49, State: Initialized, Role: FOLLOWER
I20260619 10:29:41.384193 29692 consensus_queue.cc:260] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } }
I20260619 10:29:41.384411 29692 ts_tablet_manager.cc:1428] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:29:41.384454 29546 heartbeater.cc:503] Master 127.9.90.126:42387 was elected leader, sending a full tablet report...
I20260619 10:29:41.385690 29695 raft_consensus.cc:348] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } }
I20260619 10:29:41.385843 29695 raft_consensus.cc:374] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:41.385883 29695 raft_consensus.cc:729] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38a37660eb14437b681f6e0ced19684, State: Initialized, Role: FOLLOWER
I20260619 10:29:41.385980 29695 consensus_queue.cc:260] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } }
I20260619 10:29:41.386193 29695 ts_tablet_manager.cc:1428] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 10:29:41.386246 29415 heartbeater.cc:503] Master 127.9.90.126:42387 was elected leader, sending a full tablet report...
I20260619 10:29:41.474713 29698 raft_consensus.cc:482] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 10:29:41.474836 29698 raft_consensus.cc:504] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } }
I20260619 10:29:41.475190 29698 leader_election.cc:288] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a38a37660eb14437b681f6e0ced19684 (127.9.90.65:46037), 75c80e028c244ff5ac1c050c39ab0683 (127.9.90.67:44995)
I20260619 10:29:41.477823 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.475469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:53305 (local address 127.9.90.65:46037)
0619 10:29:41.475511 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.475512 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.475521 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:29:41.475548 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.475549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.475558 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.475599 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:41.475742 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.476224 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.477133 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.477498 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.477555 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.477566 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.477693 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.477728 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:29:41.477823 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.475360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:46037 (local address 127.9.90.66:53305)
0619 10:29:41.475407 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:41.475419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:41.475466 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:41.475613 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:41.475614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:41.475615 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:41.475685 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.475687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.476297 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.476298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.477054 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.477056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.477569 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.477570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.477635 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.477638 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:41.477688 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 10:29:41.477725 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":28}
I20260619 10:29:41.477911 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.475437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.90.66:37539 (local address 127.9.90.67:44995)
0619 10:29:41.475487 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.475489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.475664 (+ 175us) server_negotiation.cc:407] Connection header received
0619 10:29:41.475714 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.475715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.475725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.475772 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 10:29:41.475900 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.476349 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.477231 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.477577 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.477633 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.477643 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.477804 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.477842 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 10:29:41.478106 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.475362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.67:44995 (local address 127.9.90.66:37539)
0619 10:29:41.475595 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:41.475604 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:29:41.475661 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:29:41.475774 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:29:41.475775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:29:41.475776 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 10:29:41.475843 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.475845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.476424 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.476424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.477143 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:29:41.477145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.477651 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.477652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:29:41.477725 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.477728 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 10:29:41.477773 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:29:41.477827 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":69,"thread_start_us":102,"threads_started":1}
I20260619 10:29:41.478122 29369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7269910d2c49b9b390a36f85ef2c32" candidate_uuid: "bca38b3f3a164d65bc4b9298f0feda49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38a37660eb14437b681f6e0ced19684" is_pre_election: true
I20260619 10:29:41.478273 29369 raft_consensus.cc:2454] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bca38b3f3a164d65bc4b9298f0feda49 in term 0.
I20260619 10:29:41.478447 29631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7269910d2c49b9b390a36f85ef2c32" candidate_uuid: "bca38b3f3a164d65bc4b9298f0feda49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c80e028c244ff5ac1c050c39ab0683" is_pre_election: true
I20260619 10:29:41.478500 29434 leader_election.cc:302] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a38a37660eb14437b681f6e0ced19684, bca38b3f3a164d65bc4b9298f0feda49; no voters:
I20260619 10:29:41.478581 29631 raft_consensus.cc:2454] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bca38b3f3a164d65bc4b9298f0feda49 in term 0.
I20260619 10:29:41.478600 29698 raft_consensus.cc:2781] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 10:29:41.478637 29698 raft_consensus.cc:482] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 10:29:41.478664 29698 raft_consensus.cc:3037] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:41.479398 29698 raft_consensus.cc:504] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } }
I20260619 10:29:41.479504 29698 leader_election.cc:288] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [CANDIDATE]: Term 1 election: Requested vote from peers a38a37660eb14437b681f6e0ced19684 (127.9.90.65:46037), 75c80e028c244ff5ac1c050c39ab0683 (127.9.90.67:44995)
I20260619 10:29:41.479679 29631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7269910d2c49b9b390a36f85ef2c32" candidate_uuid: "bca38b3f3a164d65bc4b9298f0feda49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c80e028c244ff5ac1c050c39ab0683"
I20260619 10:29:41.479677 29369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7269910d2c49b9b390a36f85ef2c32" candidate_uuid: "bca38b3f3a164d65bc4b9298f0feda49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38a37660eb14437b681f6e0ced19684"
I20260619 10:29:41.479754 29369 raft_consensus.cc:3037] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:41.479754 29631 raft_consensus.cc:3037] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:41.480592 29631 raft_consensus.cc:2454] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bca38b3f3a164d65bc4b9298f0feda49 in term 1.
I20260619 10:29:41.480592 29369 raft_consensus.cc:2454] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bca38b3f3a164d65bc4b9298f0feda49 in term 1.
I20260619 10:29:41.480780 29435 leader_election.cc:302] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75c80e028c244ff5ac1c050c39ab0683, bca38b3f3a164d65bc4b9298f0feda49; no voters:
I20260619 10:29:41.480865 29698 raft_consensus.cc:2781] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:41.481024 29698 raft_consensus.cc:686] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 1 LEADER]: Becoming Leader. State: Replica: bca38b3f3a164d65bc4b9298f0feda49, State: Running, Role: LEADER
I20260619 10:29:41.481110 29698 consensus_queue.cc:237] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } }
I20260619 10:29:41.481945 29226 catalog_manager.cc:5495] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 reported cstate change: term changed from 0 to 1, leader changed from <none> to bca38b3f3a164d65bc4b9298f0feda49 (127.9.90.66). New cstate: current_term: 1 leader_uuid: "bca38b3f3a164d65bc4b9298f0feda49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bca38b3f3a164d65bc4b9298f0feda49" member_type: VOTER last_known_addr { host: "127.9.90.66" port: 46013 } health_report { overall_health: HEALTHY } } }
W20260619 10:29:41.485479 29547 tablet.cc:2369] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 10:29:41.489205 29689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.486664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.9.90.66:46013)
0619 10:29:41.486717 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:41.486719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:41.486731 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:29:41.486813 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:41.486813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:41.486831 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:41.486886 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:29:41.486998 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.487456 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.488474 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:41.488666 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:41.488719 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:41.488863 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:29:41.488981 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:29:41.489052 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:41.489089 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 10:29:41.489107 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:29:41.491665 29631 raft_consensus.cc:1261] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [term 1 FOLLOWER]: Refusing update from remote peer bca38b3f3a164d65bc4b9298f0feda49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:41.491665 29369 raft_consensus.cc:1261] T ce7269910d2c49b9b390a36f85ef2c32 P a38a37660eb14437b681f6e0ced19684 [term 1 FOLLOWER]: Refusing update from remote peer bca38b3f3a164d65bc4b9298f0feda49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:29:41.492012 29701 consensus_queue.cc:1035] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a38a37660eb14437b681f6e0ced19684" member_type: VOTER last_known_addr { host: "127.9.90.65" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:41.492134 29698 consensus_queue.cc:1035] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c80e028c244ff5ac1c050c39ab0683" member_type: VOTER last_known_addr { host: "127.9.90.67" port: 44995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:29:41.494882 29706 mvcc.cc:205] Tried to move back new op lower bound from 7298518964187627520 to 7298518964147290112. Current Snapshot: MvccSnapshot[applied={T|T < 7298518964187627520}]
I20260619 10:29:41.496464 29226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:41.496552 29226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 10:29:41.497293 29226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c10143301e3d4ba59d4492f301f86a28: Sending DeleteTablet for 3 replicas of tablet ce7269910d2c49b9b390a36f85ef2c32
I20260619 10:29:41.497639 29480 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7269910d2c49b9b390a36f85ef2c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:41 UTC) from {username='slave'} at 127.0.0.1:60666
I20260619 10:29:41.497655 29611 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7269910d2c49b9b390a36f85ef2c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:41 UTC) from {username='slave'} at 127.0.0.1:59188
I20260619 10:29:41.497860 29713 tablet_replica.cc:330] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: stopping tablet replica
I20260619 10:29:41.497877 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 29287
I20260619 10:29:41.497910 29349 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7269910d2c49b9b390a36f85ef2c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 10:29:41 UTC) from {username='slave'} at 127.0.0.1:35936
I20260619 10:29:41.497954 29713 raft_consensus.cc:2229] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 1 LEADER]: Raft consensus shutting down.
I20260619 10:29:41.498024 29713 raft_consensus.cc:2258] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:41.498734 29714 tablet_replica.cc:330] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: stopping tablet replica
I20260619 10:29:41.498895 29714 raft_consensus.cc:2229] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 10:29:41.498926 29713 ts_tablet_manager.cc:1905] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:41.499059 29714 raft_consensus.cc:2258] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 10:29:41.499483 29714 ts_tablet_manager.cc:1905] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 10:29:41.500813 29713 ts_tablet_manager.cc:1918] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:41.500923 29713 log.cc:1196] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-1/wal/wals/ce7269910d2c49b9b390a36f85ef2c32
I20260619 10:29:41.501108 29714 ts_tablet_manager.cc:1918] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 10:29:41.501222 29714 log.cc:1196] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: Deleting WAL directory at /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781864754199919-9577-0/minicluster-data/ts-2/wal/wals/ce7269910d2c49b9b390a36f85ef2c32
I20260619 10:29:41.501295 29713 ts_tablet_manager.cc:1939] T ce7269910d2c49b9b390a36f85ef2c32 P bca38b3f3a164d65bc4b9298f0feda49: Deleting consensus metadata
I20260619 10:29:41.501519 29714 ts_tablet_manager.cc:1939] T ce7269910d2c49b9b390a36f85ef2c32 P 75c80e028c244ff5ac1c050c39ab0683: Deleting consensus metadata
I20260619 10:29:41.501684 29214 catalog_manager.cc:4841] TS bca38b3f3a164d65bc4b9298f0feda49 (127.9.90.66:46013): tablet ce7269910d2c49b9b390a36f85ef2c32 (table test-table [id=46030e168da24eb79a5d987492ecd511]) successfully deleted
I20260619 10:29:41.501992 29213 catalog_manager.cc:4841] TS 75c80e028c244ff5ac1c050c39ab0683 (127.9.90.67:44995): tablet ce7269910d2c49b9b390a36f85ef2c32 (table test-table [id=46030e168da24eb79a5d987492ecd511]) successfully deleted
W20260619 10:29:41.504639 29212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.90.65:46037 (error 108)
I20260619 10:29:41.504873 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 29418
I20260619 10:29:41.505393 29685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:41.504966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.65:46037 (local address 127.0.0.1:35944)
0619 10:29:41.505322 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 10:29:41.505360 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.65:46037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339}
W20260619 10:29:41.505848 29212 catalog_manager.cc:4584] TS a38a37660eb14437b681f6e0ced19684 (127.9.90.65:46037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce7269910d2c49b9b390a36f85ef2c32: Network error: Client connection negotiation failed: client connection to 127.9.90.65:46037: connect: Connection refused (error 111)
I20260619 10:29:41.508891 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 29549
I20260619 10:29:41.512786 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 29196
2026-06-19T10:29:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (705 ms)
[----------] 6 tests from EncryptionPolicyTest (7600 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:41 dist-test-slave-hw7q krb5kdc[29718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:41 dist-test-slave-hw7q krb5kdc[29718](info): set up 2 sockets
Jun 19 10:29:41 dist-test-slave-hw7q krb5kdc[29718](info): commencing operation
krb5kdc: starting...
W20260619 10:29:42.461565 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:42 dist-test-slave-hw7q krb5kdc[29718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:42.570638 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:42.647224 29731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:42.647434 29731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:42.647456 29731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:42.648564 29731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:42.648602 29731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:42.648615 29731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:42.648627 29731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:42.648638 29731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:42.650048 29731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:42.650249 29731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:42.650420 29731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:42.654863 29737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:42.654928 29736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:42.655005 29731 server_base.cc:1034] running on GCE node
W20260619 10:29:42.654881 29739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:42.655313 29731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:42.655596 29731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:42.656749 29731 hybrid_clock.cc:648] HybridClock initialized: now 1781864982656739 us; error 41 us; skew 500 ppm
Jun 19 10:29:42 dist-test-slave-hw7q krb5kdc[29718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:42.659067 29731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:42.659447 29731 webserver.cc:460] Webserver started at http://127.0.0.1:42765/ using document root <none> and password file <none>
I20260619 10:29:42.659632 29731 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:42.659679 29731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:42.659832 29731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:42.660841 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "b485f5dc02fd455f9ec45cd2a0e9de69"
format_stamp: "Formatted at 2026-06-19 10:29:42 on dist-test-slave-hw7q"
server_key: "ea37453c6b6e635c02fa9b90aeb627ea"
server_key_iv: "0e8b2d5a7dbfbee9917d61a0a654af6c"
server_key_version: "encryptionkey@0"
I20260619 10:29:42.661159 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "b485f5dc02fd455f9ec45cd2a0e9de69"
format_stamp: "Formatted at 2026-06-19 10:29:42 on dist-test-slave-hw7q"
server_key: "ea37453c6b6e635c02fa9b90aeb627ea"
server_key_iv: "0e8b2d5a7dbfbee9917d61a0a654af6c"
server_key_version: "encryptionkey@0"
I20260619 10:29:42.662405 29731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:29:42.663029 29746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:42.663166 29731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:42.663244 29731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "b485f5dc02fd455f9ec45cd2a0e9de69"
format_stamp: "Formatted at 2026-06-19 10:29:42 on dist-test-slave-hw7q"
server_key: "ea37453c6b6e635c02fa9b90aeb627ea"
server_key_iv: "0e8b2d5a7dbfbee9917d61a0a654af6c"
server_key_version: "encryptionkey@0"
I20260619 10:29:42.663302 29731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:42.679633 29731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:42.681764 29731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:42.681914 29731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:42.685495 29731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41581
I20260619 10:29:42.685546 29798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41581 every 8 connection(s)
I20260619 10:29:42.685883 29731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:42.686861 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:42.689060 29799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69: Bootstrap starting.
I20260619 10:29:42.689584 29799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:42.689888 29799 log.cc:824] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:42.690536 29799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69: No bootstrap required, opened a new log
I20260619 10:29:42.692086 29799 raft_consensus.cc:348] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b485f5dc02fd455f9ec45cd2a0e9de69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } }
I20260619 10:29:42.692222 29799 raft_consensus.cc:374] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:42.692255 29799 raft_consensus.cc:729] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b485f5dc02fd455f9ec45cd2a0e9de69, State: Initialized, Role: FOLLOWER
I20260619 10:29:42.692389 29799 consensus_queue.cc:260] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b485f5dc02fd455f9ec45cd2a0e9de69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } }
I20260619 10:29:42.692466 29799 raft_consensus.cc:388] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:42.692495 29799 raft_consensus.cc:482] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:42.692541 29799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:42.693032 29799 raft_consensus.cc:504] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b485f5dc02fd455f9ec45cd2a0e9de69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } }
I20260619 10:29:42.693123 29799 leader_election.cc:302] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b485f5dc02fd455f9ec45cd2a0e9de69; no voters:
I20260619 10:29:42.693281 29799 leader_election.cc:288] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:42.693312 29802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:42.693451 29802 raft_consensus.cc:686] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [term 1 LEADER]: Becoming Leader. State: Replica: b485f5dc02fd455f9ec45cd2a0e9de69, State: Running, Role: LEADER
I20260619 10:29:42.693514 29799 sys_catalog.cc:564] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:42.693538 29802 consensus_queue.cc:237] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b485f5dc02fd455f9ec45cd2a0e9de69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } }
I20260619 10:29:42.693887 29804 sys_catalog.cc:455] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b485f5dc02fd455f9ec45cd2a0e9de69. Latest consensus state: current_term: 1 leader_uuid: "b485f5dc02fd455f9ec45cd2a0e9de69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b485f5dc02fd455f9ec45cd2a0e9de69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } } }
I20260619 10:29:42.693967 29804 sys_catalog.cc:458] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:42.694064 29803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b485f5dc02fd455f9ec45cd2a0e9de69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b485f5dc02fd455f9ec45cd2a0e9de69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } } }
I20260619 10:29:42.694146 29803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:42.694166 29809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:42.694563 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 29731
I20260619 10:29:42.694664 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:42.694684 29809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:42.694866 9577 external_mini_cluster.cc:1428] Setting key c01d6f164144497628d0b1ba849c0dc0
I20260619 10:29:42.696439 29809 catalog_manager.cc:1347] Generated new cluster ID: ecc4a4fb26774544a4f286ae7188e0b0
I20260619 10:29:42.696489 29809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:29:42 dist-test-slave-hw7q krb5kdc[29718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 10:29:42.706388 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:42.695309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59481 (local address 127.0.0.1:41581)
0619 10:29:42.695605 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:42.695613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:42.695640 (+ 27us) server_negotiation.cc:407] Connection header received
0619 10:29:42.695770 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:42.695777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:42.695824 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:42.695936 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:42.696694 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:42.698135 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:42.699137 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:42.699776 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:42.699883 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:42.702865 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:42.702869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:42.702870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:42.702881 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:42.704642 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:42.705069 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:42.705072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:42.705072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:42.705201 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:42.705428 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:42.705429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:42.705429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:42.705724 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:42.705898 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:42.706052 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 10:29:42.706152 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":62,"threads_started":1}
I20260619 10:29:42.707528 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:42.716647 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:42.710032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38255 (local address 127.0.0.1:41581)
0619 10:29:42.710067 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:42.710068 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:42.710203 (+ 135us) server_negotiation.cc:407] Connection header received
0619 10:29:42.710271 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:42.710272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:42.710281 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:42.710327 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:42.710836 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:42.711615 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:42.712508 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:42.712823 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:42.712885 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:42.714136 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:42.714137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:42.714137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:42.714144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:42.715536 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:42.715910 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:42.715911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:42.715911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:42.715964 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:42.716155 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:42.716156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:42.716156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:42.716277 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:42.716369 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:42.716502 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 10:29:42.716558 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:29:42.717381 29809 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:42.718096 29809 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:42.725543 29809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b485f5dc02fd455f9ec45cd2a0e9de69: Generated new TSK 0
I20260619 10:29:42.725732 29809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:42.731326 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 29731
W20260619 10:29:42.734890 29716 connection.cc:535] client connection to 127.0.0.1:41581 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41581): Connection reset by peer (error 104)
2026-06-19T10:29:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:42 dist-test-slave-hw7q krb5kdc[29830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:42 dist-test-slave-hw7q krb5kdc[29830](info): set up 2 sockets
Jun 19 10:29:42 dist-test-slave-hw7q krb5kdc[29830](info): commencing operation
krb5kdc: starting...
W20260619 10:29:43.681542 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:43 dist-test-slave-hw7q krb5kdc[29830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:43.788250 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:43.865167 29843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:43.865373 29843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:43.865402 29843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:43.866533 29843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:43.866575 29843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:43.866593 29843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:43.866609 29843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:43.866623 29843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:43.868049 29843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:43.868278 29843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:43.868472 29843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:43.872869 29849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:43.872983 29843 server_base.cc:1034] running on GCE node
W20260619 10:29:43.872874 29851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:43.872874 29848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:43.873353 29843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:43.873634 29843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:43.874779 29843 hybrid_clock.cc:648] HybridClock initialized: now 1781864983874757 us; error 36 us; skew 500 ppm
Jun 19 10:29:43 dist-test-slave-hw7q krb5kdc[29830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:43.877064 29843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:43.877468 29843 webserver.cc:460] Webserver started at http://127.0.0.1:34063/ using document root <none> and password file <none>
I20260619 10:29:43.877677 29843 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:43.877727 29843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:43.877861 29843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:43.878890 29843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "82fe24667c19434f9c203ab27a85146a"
format_stamp: "Formatted at 2026-06-19 10:29:43 on dist-test-slave-hw7q"
server_key: "9ce3e446ca6d879b1e893496953c63ce"
server_key_iv: "4224c2fb74ccfb9bbaa00aeeb6d5b3b2"
server_key_version: "encryptionkey@0"
I20260619 10:29:43.879199 29843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "82fe24667c19434f9c203ab27a85146a"
format_stamp: "Formatted at 2026-06-19 10:29:43 on dist-test-slave-hw7q"
server_key: "9ce3e446ca6d879b1e893496953c63ce"
server_key_iv: "4224c2fb74ccfb9bbaa00aeeb6d5b3b2"
server_key_version: "encryptionkey@0"
I20260619 10:29:43.880419 29843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:29:43.881102 29858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:43.881340 29843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:29:43.881430 29843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "82fe24667c19434f9c203ab27a85146a"
format_stamp: "Formatted at 2026-06-19 10:29:43 on dist-test-slave-hw7q"
server_key: "9ce3e446ca6d879b1e893496953c63ce"
server_key_iv: "4224c2fb74ccfb9bbaa00aeeb6d5b3b2"
server_key_version: "encryptionkey@0"
I20260619 10:29:43.881505 29843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:43.899713 29843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:43.901978 29843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:43.902124 29843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:43.906186 29843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38111
I20260619 10:29:43.906219 29910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38111 every 8 connection(s)
I20260619 10:29:43.906550 29843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:43.907485 29911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:43.909603 29911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a: Bootstrap starting.
I20260619 10:29:43.910128 29911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:43.910355 29911 log.cc:824] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:43.910936 29911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a: No bootstrap required, opened a new log
I20260619 10:29:43.912283 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 29843
I20260619 10:29:43.912384 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:43.912456 29911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fe24667c19434f9c203ab27a85146a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260619 10:29:43.912569 9577 external_mini_cluster.cc:1428] Setting key b6c9ce6ce047adb134a31ebcbf1649e4
I20260619 10:29:43.912596 29911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:43.912618 29911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82fe24667c19434f9c203ab27a85146a, State: Initialized, Role: FOLLOWER
I20260619 10:29:43.912753 29911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fe24667c19434f9c203ab27a85146a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260619 10:29:43.912811 29911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:43.912830 29911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:43.912874 29911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:43.913461 29911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fe24667c19434f9c203ab27a85146a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260619 10:29:43.913553 29911 leader_election.cc:302] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82fe24667c19434f9c203ab27a85146a; no voters:
I20260619 10:29:43.913705 29911 leader_election.cc:288] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:43.913725 29916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:43.913862 29916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [term 1 LEADER]: Becoming Leader. State: Replica: 82fe24667c19434f9c203ab27a85146a, State: Running, Role: LEADER
I20260619 10:29:43.913949 29911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:43.913947 29916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fe24667c19434f9c203ab27a85146a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260619 10:29:43.914433 29917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82fe24667c19434f9c203ab27a85146a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fe24667c19434f9c203ab27a85146a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } } }
I20260619 10:29:43.914451 29918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82fe24667c19434f9c203ab27a85146a. Latest consensus state: current_term: 1 leader_uuid: "82fe24667c19434f9c203ab27a85146a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fe24667c19434f9c203ab27a85146a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } } }
I20260619 10:29:43.914537 29917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:43.914543 29918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:43.914804 29925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:43.915328 29925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:43.916661 29925 catalog_manager.cc:1347] Generated new cluster ID: a8ac6bdf9b26439aafdcaff80c2a19c5
I20260619 10:29:43.916719 29925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:43.919862 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:43.913176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54757 (local address 127.0.0.1:38111)
0619 10:29:43.913633 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:43.913642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:43.913672 (+ 30us) server_negotiation.cc:407] Connection header received
0619 10:29:43.913807 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:43.913814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:43.913873 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:43.914059 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:43.915335 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:43.916696 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:43.917771 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:43.918422 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:43.918531 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:43.919213 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:43.919216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:43.919217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:43.919414 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:43.919450 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:43.919466 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:43.919545 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:43.919634 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 10:29:43.919660 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":369,"threads_started":1}
I20260619 10:29:43.920428 29925 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:43.920747 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:43.921046 29925 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:43.923548 29925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82fe24667c19434f9c203ab27a85146a: Generated new TSK 0
I20260619 10:29:43.923691 29925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:43.925375 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:43.922371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58481 (local address 127.0.0.1:38111)
0619 10:29:43.922401 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:43.922402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:43.922409 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:29:43.922427 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:43.922428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:43.922437 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:43.922500 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:43.922603 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:43.923374 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:43.924194 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:43.924502 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:43.924572 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:43.925156 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:43.925157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:43.925157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:43.925162 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:43.925175 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:43.925180 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:43.925242 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:43.925295 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 10:29:43.925306 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:29:43.927009 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 29843
W20260619 10:29:43.930787 29828 connection.cc:535] client connection to 127.0.0.1:38111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38111): Connection reset by peer (error 104)
2026-06-19T10:29:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:43 dist-test-slave-hw7q krb5kdc[29942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:43 dist-test-slave-hw7q krb5kdc[29942](info): set up 2 sockets
Jun 19 10:29:43 dist-test-slave-hw7q krb5kdc[29942](info): commencing operation
krb5kdc: starting...
W20260619 10:29:44.861439 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:44 dist-test-slave-hw7q krb5kdc[29942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:44.969682 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:45.048693 29955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:45.048893 29955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:45.048923 29955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:45.050048 29955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:45.050084 29955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:45.050104 29955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:45.050120 29955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:45.050136 29955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:45.051537 29955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:45.051766 29955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:45.051950 29955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:45.056226 29960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:45.056259 29961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:45.056444 29955 server_base.cc:1034] running on GCE node
W20260619 10:29:45.056259 29963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:45.056718 29955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:45.056990 29955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:45.058135 29955 hybrid_clock.cc:648] HybridClock initialized: now 1781864985058107 us; error 43 us; skew 500 ppm
Jun 19 10:29:45 dist-test-slave-hw7q krb5kdc[29942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:45.063058 29955 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:45.063432 29955 webserver.cc:460] Webserver started at http://127.0.0.1:40833/ using document root <none> and password file <none>
I20260619 10:29:45.063604 29955 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:45.063642 29955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:45.063735 29955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:45.064652 29955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "c0f6477a52cb426c8903e5c0107b7dd2"
format_stamp: "Formatted at 2026-06-19 10:29:45 on dist-test-slave-hw7q"
server_key: "2dcaaea351e108eccd591f8ef11aac3b"
server_key_iv: "8f6e3b4b9a4b6e83d1ae2637866b698f"
server_key_version: "encryptionkey@0"
I20260619 10:29:45.064953 29955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "c0f6477a52cb426c8903e5c0107b7dd2"
format_stamp: "Formatted at 2026-06-19 10:29:45 on dist-test-slave-hw7q"
server_key: "2dcaaea351e108eccd591f8ef11aac3b"
server_key_iv: "8f6e3b4b9a4b6e83d1ae2637866b698f"
server_key_version: "encryptionkey@0"
I20260619 10:29:45.066063 29955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:29:45.066669 29970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:45.066815 29955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:45.066892 29955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "c0f6477a52cb426c8903e5c0107b7dd2"
format_stamp: "Formatted at 2026-06-19 10:29:45 on dist-test-slave-hw7q"
server_key: "2dcaaea351e108eccd591f8ef11aac3b"
server_key_iv: "8f6e3b4b9a4b6e83d1ae2637866b698f"
server_key_version: "encryptionkey@0"
I20260619 10:29:45.066948 29955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:45.069973 29955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:45.071911 29955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:45.072031 29955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:45.075594 29955 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46045
I20260619 10:29:45.075698 30022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46045 every 8 connection(s)
I20260619 10:29:45.075928 29955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:45.076857 30023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:45.078979 30023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2: Bootstrap starting.
I20260619 10:29:45.079474 30023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:45.079684 30023 log.cc:824] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:45.080289 30023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2: No bootstrap required, opened a new log
I20260619 10:29:45.081773 30023 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f6477a52cb426c8903e5c0107b7dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260619 10:29:45.081954 30023 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:45.081979 30023 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f6477a52cb426c8903e5c0107b7dd2, State: Initialized, Role: FOLLOWER
I20260619 10:29:45.082079 30023 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f6477a52cb426c8903e5c0107b7dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260619 10:29:45.082129 30023 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:45.082147 30023 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:45.082178 30023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:45.082633 30023 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f6477a52cb426c8903e5c0107b7dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260619 10:29:45.082712 30023 leader_election.cc:302] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0f6477a52cb426c8903e5c0107b7dd2; no voters:
I20260619 10:29:45.082831 30023 leader_election.cc:288] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:45.082886 30026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:45.083009 30026 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [term 1 LEADER]: Becoming Leader. State: Replica: c0f6477a52cb426c8903e5c0107b7dd2, State: Running, Role: LEADER
I20260619 10:29:45.083078 30023 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:45.083122 30026 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f6477a52cb426c8903e5c0107b7dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260619 10:29:45.083384 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 29955
I20260619 10:29:45.083505 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:45.083498 30027 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0f6477a52cb426c8903e5c0107b7dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f6477a52cb426c8903e5c0107b7dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260619 10:29:45.083581 30027 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:45.083719 9577 external_mini_cluster.cc:1428] Setting key 07e084897bcb22c6e77335a4db308611
I20260619 10:29:45.083741 30028 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0f6477a52cb426c8903e5c0107b7dd2. Latest consensus state: current_term: 1 leader_uuid: "c0f6477a52cb426c8903e5c0107b7dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f6477a52cb426c8903e5c0107b7dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260619 10:29:45.083801 30028 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:45.083921 30030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:45.084390 30030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:45.086062 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:45.084283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59941 (local address 127.0.0.1:46045)
0619 10:29:45.084428 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:45.084486 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:45.084520 (+ 34us) server_negotiation.cc:407] Connection header received
0619 10:29:45.084656 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:45.084664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:45.084696 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:45.084814 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:45.085607 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:45.085611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:45.085611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:45.085615 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:45.085651 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:45.085662 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:45.085707 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:45.085847 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 10:29:45.085892 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260619 10:29:45.087005 30030 catalog_manager.cc:1347] Generated new cluster ID: 1cfe5c87dfcd4c2ab9079f298aa2dac4
I20260619 10:29:45.087051 30030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:45.087150 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:45.090868 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:45.088508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55685 (local address 127.0.0.1:46045)
0619 10:29:45.088537 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:45.088538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:45.089943 (+ 1405us) server_negotiation.cc:407] Connection header received
0619 10:29:45.090030 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:45.090031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:45.090036 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:45.090098 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:45.090670 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:45.090671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:45.090671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:45.090675 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:45.090691 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:45.090695 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:45.090755 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:45.090794 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 10:29:45.090807 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:29:45.097321 30030 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:45.097465 30030 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:45.100173 30030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0f6477a52cb426c8903e5c0107b7dd2: Generated new TSK 0
I20260619 10:29:45.100353 30030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:45.105893 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 29955
2026-06-19T10:29:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:45 dist-test-slave-hw7q krb5kdc[30054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:45 dist-test-slave-hw7q krb5kdc[30054](info): set up 2 sockets
Jun 19 10:29:45 dist-test-slave-hw7q krb5kdc[30054](info): commencing operation
krb5kdc: starting...
W20260619 10:29:46.049480 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:46 dist-test-slave-hw7q krb5kdc[30054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:46.154306 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:46.231323 30067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:46.231518 30067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:46.231547 30067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:46.232659 30067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:46.232702 30067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:46.232720 30067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:46.232736 30067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:46.232751 30067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:46.234187 30067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:46.234416 30067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:46.234609 30067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:46.239084 30075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:46.239096 30072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:46.239091 30073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:46.239413 30067 server_base.cc:1034] running on GCE node
I20260619 10:29:46.239645 30067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:46.239902 30067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:46.241034 30067 hybrid_clock.cc:648] HybridClock initialized: now 1781864986241014 us; error 29 us; skew 500 ppm
Jun 19 10:29:46 dist-test-slave-hw7q krb5kdc[30054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:46.243309 30067 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:46.243697 30067 webserver.cc:460] Webserver started at http://127.0.0.1:36399/ using document root <none> and password file <none>
I20260619 10:29:46.243901 30067 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:46.243947 30067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:46.244067 30067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:46.245044 30067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "fa4b844f252a450db9246445896a9c7f"
format_stamp: "Formatted at 2026-06-19 10:29:46 on dist-test-slave-hw7q"
server_key: "42c73c6af43322eadc15c418f2b599ca"
server_key_iv: "0aa917c7faef89a510203f4f1b15b316"
server_key_version: "encryptionkey@0"
I20260619 10:29:46.245370 30067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "fa4b844f252a450db9246445896a9c7f"
format_stamp: "Formatted at 2026-06-19 10:29:46 on dist-test-slave-hw7q"
server_key: "42c73c6af43322eadc15c418f2b599ca"
server_key_iv: "0aa917c7faef89a510203f4f1b15b316"
server_key_version: "encryptionkey@0"
I20260619 10:29:46.246829 30067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:29:46.247499 30082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:46.247654 30067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:46.247737 30067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "fa4b844f252a450db9246445896a9c7f"
format_stamp: "Formatted at 2026-06-19 10:29:46 on dist-test-slave-hw7q"
server_key: "42c73c6af43322eadc15c418f2b599ca"
server_key_iv: "0aa917c7faef89a510203f4f1b15b316"
server_key_version: "encryptionkey@0"
I20260619 10:29:46.247797 30067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:46.265848 30067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:46.267392 30067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:46.267518 30067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:46.271071 30067 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39219
I20260619 10:29:46.271108 30134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39219 every 8 connection(s)
I20260619 10:29:46.271401 30067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:46.272277 30135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:46.274405 30135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f: Bootstrap starting.
I20260619 10:29:46.274904 30135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:46.275141 30135 log.cc:824] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:46.275758 30135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f: No bootstrap required, opened a new log
I20260619 10:29:46.277268 30135 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4b844f252a450db9246445896a9c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39219 } }
I20260619 10:29:46.277398 30135 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:46.277429 30135 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4b844f252a450db9246445896a9c7f, State: Initialized, Role: FOLLOWER
I20260619 10:29:46.277565 30135 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4b844f252a450db9246445896a9c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39219 } }
I20260619 10:29:46.277645 30135 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:46.277683 30135 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:46.277729 30135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:46.278223 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 30067
I20260619 10:29:46.278278 30135 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4b844f252a450db9246445896a9c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39219 } }
I20260619 10:29:46.278335 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:46.278378 30135 leader_election.cc:302] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa4b844f252a450db9246445896a9c7f; no voters:
I20260619 10:29:46.278518 30135 leader_election.cc:288] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:46.278553 9577 external_mini_cluster.cc:1428] Setting key 68ed1640de1908c0f63fee32d89fb3e0
I20260619 10:29:46.278553 30138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:46.278657 30138 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [term 1 LEADER]: Becoming Leader. State: Replica: fa4b844f252a450db9246445896a9c7f, State: Running, Role: LEADER
I20260619 10:29:46.278759 30135 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:46.278805 30138 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4b844f252a450db9246445896a9c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39219 } }
I20260619 10:29:46.279317 30140 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa4b844f252a450db9246445896a9c7f. Latest consensus state: current_term: 1 leader_uuid: "fa4b844f252a450db9246445896a9c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4b844f252a450db9246445896a9c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39219 } } }
I20260619 10:29:46.279385 30140 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:46.279357 30139 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa4b844f252a450db9246445896a9c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4b844f252a450db9246445896a9c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39219 } } }
I20260619 10:29:46.279451 30139 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:46.279770 30145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:46.280552 30145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:46.282058 30145 catalog_manager.cc:1347] Generated new cluster ID: 4be5f9f83bb7415baf3a716a39d67361
I20260619 10:29:46.282100 30145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:29:46 dist-test-slave-hw7q krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 10:29:46.288373 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:46.278994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47615 (local address 127.0.0.1:39219)
0619 10:29:46.279151 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:46.279162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:46.279227 (+ 65us) server_negotiation.cc:407] Connection header received
0619 10:29:46.279399 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:46.279406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:46.279473 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:46.280230 (+ 757us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:46.280476 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:46.281651 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:46.282466 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:46.282887 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:46.282989 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:46.285195 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:46.285199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:46.285200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:46.285212 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:46.286858 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:46.287206 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:46.287209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:46.287209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:46.287317 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:46.287504 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:46.287505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:46.287505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:46.287746 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:46.287893 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:46.288005 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 10:29:46.288230 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260619 10:29:46.289222 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:46.294142 30145 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:46.294780 30145 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:46.299579 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:46.293935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36995 (local address 127.0.0.1:39219)
0619 10:29:46.293978 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:46.293979 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:46.294122 (+ 143us) server_negotiation.cc:407] Connection header received
0619 10:29:46.294211 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:46.294212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:46.294223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:46.294279 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:46.294782 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:46.295249 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:46.295901 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:46.296111 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:46.296170 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:46.297247 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:46.297249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:46.297249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:46.297256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:46.298577 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:46.298879 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:46.298880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:46.298880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:46.298933 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:46.299103 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:46.299104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:46.299104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:46.299207 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:46.299291 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:46.299407 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 10:29:46.299496 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:29:46.300843 30145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa4b844f252a450db9246445896a9c7f: Generated new TSK 0
I20260619 10:29:46.301028 30145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:46.307703 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 30067
2026-06-19T10:29:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:46 dist-test-slave-hw7q krb5kdc[30166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:46 dist-test-slave-hw7q krb5kdc[30166](info): set up 2 sockets
Jun 19 10:29:46 dist-test-slave-hw7q krb5kdc[30166](info): commencing operation
krb5kdc: starting...
W20260619 10:29:47.253487 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:47 dist-test-slave-hw7q krb5kdc[30166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:47.360628 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:47.437566 30179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:47.437932 30179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:47.437988 30179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:47.439164 30179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:47.439208 30179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:47.439220 30179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:47.439242 30179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:47.439262 30179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:47.440742 30179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:47.440955 30179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:47.441155 30179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:29:47.445688 30179 server_base.cc:1034] running on GCE node
W20260619 10:29:47.445641 30187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:47.445659 30184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:47.445672 30185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:47.446105 30179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:47.446380 30179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:47.447517 30179 hybrid_clock.cc:648] HybridClock initialized: now 1781864987447504 us; error 29 us; skew 500 ppm
Jun 19 10:29:47 dist-test-slave-hw7q krb5kdc[30166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:47.449847 30179 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:47.450232 30179 webserver.cc:460] Webserver started at http://127.0.0.1:42885/ using document root <none> and password file <none>
I20260619 10:29:47.450433 30179 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:47.450479 30179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:47.450600 30179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:47.451586 30179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "f9a66bea96cc465eb65a8de16fe7efa4"
format_stamp: "Formatted at 2026-06-19 10:29:47 on dist-test-slave-hw7q"
server_key: "38608000dc9f131bbc82b029b4f363ea"
server_key_iv: "34d6f71f49b5f23c5cbc4b21fea4bdc4"
server_key_version: "encryptionkey@0"
I20260619 10:29:47.451905 30179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "f9a66bea96cc465eb65a8de16fe7efa4"
format_stamp: "Formatted at 2026-06-19 10:29:47 on dist-test-slave-hw7q"
server_key: "38608000dc9f131bbc82b029b4f363ea"
server_key_iv: "34d6f71f49b5f23c5cbc4b21fea4bdc4"
server_key_version: "encryptionkey@0"
I20260619 10:29:47.453034 30179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:29:47.453615 30194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:47.453765 30179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:47.453871 30179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "f9a66bea96cc465eb65a8de16fe7efa4"
format_stamp: "Formatted at 2026-06-19 10:29:47 on dist-test-slave-hw7q"
server_key: "38608000dc9f131bbc82b029b4f363ea"
server_key_iv: "34d6f71f49b5f23c5cbc4b21fea4bdc4"
server_key_version: "encryptionkey@0"
I20260619 10:29:47.453927 30179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:47.464361 30179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:47.466311 30179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:47.466439 30179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:47.469961 30179 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40809
I20260619 10:29:47.469990 30246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40809 every 8 connection(s)
I20260619 10:29:47.470289 30179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:47.471148 30247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:47.473271 30247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4: Bootstrap starting.
I20260619 10:29:47.473759 30247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:47.474032 30247 log.cc:824] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:47.474640 30247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4: No bootstrap required, opened a new log
I20260619 10:29:47.474763 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 30179
I20260619 10:29:47.474876 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:47.475052 9577 external_mini_cluster.cc:1428] Setting key 124aaa2af6b5393196a89a039ed949c0
I20260619 10:29:47.476480 30247 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a66bea96cc465eb65a8de16fe7efa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } }
I20260619 10:29:47.476608 30247 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:47.476637 30247 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a66bea96cc465eb65a8de16fe7efa4, State: Initialized, Role: FOLLOWER
I20260619 10:29:47.476784 30247 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a66bea96cc465eb65a8de16fe7efa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } }
I20260619 10:29:47.476857 30247 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:47.476909 30247 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:47.476959 30247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:47.477439 30247 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a66bea96cc465eb65a8de16fe7efa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } }
I20260619 10:29:47.477528 30247 leader_election.cc:302] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9a66bea96cc465eb65a8de16fe7efa4; no voters:
I20260619 10:29:47.477686 30247 leader_election.cc:288] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:47.477717 30252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:47.477828 30252 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [term 1 LEADER]: Becoming Leader. State: Replica: f9a66bea96cc465eb65a8de16fe7efa4, State: Running, Role: LEADER
I20260619 10:29:47.477957 30247 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:47.477959 30252 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a66bea96cc465eb65a8de16fe7efa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } }
I20260619 10:29:47.478312 30252 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a66bea96cc465eb65a8de16fe7efa4. Latest consensus state: current_term: 1 leader_uuid: "f9a66bea96cc465eb65a8de16fe7efa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a66bea96cc465eb65a8de16fe7efa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } } }
I20260619 10:29:47.478510 30252 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:47.478446 30253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9a66bea96cc465eb65a8de16fe7efa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a66bea96cc465eb65a8de16fe7efa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } } }
I20260619 10:29:47.478868 30253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:47.479048 30261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:47.479512 30261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:47.481318 30261 catalog_manager.cc:1347] Generated new cluster ID: 46fbbd4ecf764139b23569ed513f731a
I20260619 10:29:47.481372 30261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:47.481573 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:47.475527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56703 (local address 127.0.0.1:40809)
0619 10:29:47.475693 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:47.475701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:47.475726 (+ 25us) server_negotiation.cc:407] Connection header received
0619 10:29:47.476489 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:47.476497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:47.476552 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:47.476698 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:47.476900 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:47.478262 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:47.479381 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:47.480135 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:47.480237 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:47.480871 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:47.480875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:47.480875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:47.481045 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:47.481074 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:47.481085 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:47.481139 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:47.481325 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 10:29:47.481390 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260619 10:29:47.482360 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:47.486349 30261 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:47.487030 30261 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:47.488704 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:47.486121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51283 (local address 127.0.0.1:40809)
0619 10:29:47.486163 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:47.486165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:47.486353 (+ 188us) server_negotiation.cc:407] Connection header received
0619 10:29:47.486414 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:47.486415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:47.486425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:47.486464 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:47.486577 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:47.487030 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:47.487693 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:47.487880 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:47.487931 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:47.488484 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:47.488485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:47.488485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:47.488489 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:47.488502 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:47.488506 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:47.488547 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:47.488589 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 10:29:47.488626 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 10:29:47.494109 30261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9a66bea96cc465eb65a8de16fe7efa4: Generated new TSK 0
I20260619 10:29:47.494292 30261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:47.498526 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 30179
2026-06-19T10:29:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1191 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:47 dist-test-slave-hw7q krb5kdc[30278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:47 dist-test-slave-hw7q krb5kdc[30278](info): set up 2 sockets
Jun 19 10:29:47 dist-test-slave-hw7q krb5kdc[30278](info): commencing operation
krb5kdc: starting...
W20260619 10:29:48.445717 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:48 dist-test-slave-hw7q krb5kdc[30278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:48.552486 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:48.628633 30291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:48.628821 30291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:48.628841 30291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:48.629922 30291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:48.629959 30291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:48.629972 30291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:48.629982 30291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:48.629997 30291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:48.631362 30291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:48.631531 30291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:48.631683 30291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:48.636112 30296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:48.636214 30291 server_base.cc:1034] running on GCE node
W20260619 10:29:48.636124 30297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:48.636129 30299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:48.636528 30291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:48.636776 30291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:48.637912 30291 hybrid_clock.cc:648] HybridClock initialized: now 1781864988637894 us; error 33 us; skew 500 ppm
Jun 19 10:29:48 dist-test-slave-hw7q krb5kdc[30278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:48.640323 30291 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:48.640815 30291 webserver.cc:460] Webserver started at http://127.0.0.1:39011/ using document root <none> and password file <none>
I20260619 10:29:48.641023 30291 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:48.641068 30291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:48.641171 30291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:48.642207 30291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019"
format_stamp: "Formatted at 2026-06-19 10:29:48 on dist-test-slave-hw7q"
server_key: "3033ea499a3891756a66dc7953f71bb6"
server_key_iv: "c37f2ecda72ef84f6d61dbc88c5020da"
server_key_version: "encryptionkey@0"
I20260619 10:29:48.642518 30291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019"
format_stamp: "Formatted at 2026-06-19 10:29:48 on dist-test-slave-hw7q"
server_key: "3033ea499a3891756a66dc7953f71bb6"
server_key_iv: "c37f2ecda72ef84f6d61dbc88c5020da"
server_key_version: "encryptionkey@0"
I20260619 10:29:48.643733 30291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:48.644390 30306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:48.644577 30291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:48.644655 30291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019"
format_stamp: "Formatted at 2026-06-19 10:29:48 on dist-test-slave-hw7q"
server_key: "3033ea499a3891756a66dc7953f71bb6"
server_key_iv: "c37f2ecda72ef84f6d61dbc88c5020da"
server_key_version: "encryptionkey@0"
I20260619 10:29:48.644707 30291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:48.647823 30291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:48.649499 30291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:48.649603 30291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:48.653013 30291 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45495
I20260619 10:29:48.653048 30358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45495 every 8 connection(s)
I20260619 10:29:48.653329 30291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:48.654223 30359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:48.656261 30359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019: Bootstrap starting.
I20260619 10:29:48.656338 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 30291
I20260619 10:29:48.656454 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:48.656615 9577 external_mini_cluster.cc:1428] Setting key 1a19c063b012bb5f404cf65379dd319c
I20260619 10:29:48.656870 30359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:48.657181 30359 log.cc:824] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:48.658007 30359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019: No bootstrap required, opened a new log
I20260619 10:29:48.659389 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:48.657041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56327 (local address 127.0.0.1:45495)
0619 10:29:48.657217 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:48.657226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:48.657562 (+ 336us) server_negotiation.cc:407] Connection header received
0619 10:29:48.658139 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:48.658148 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:48.658192 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:48.658336 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:48.658913 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:48.658916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:48.658917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:48.658920 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:48.658959 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:48.658973 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:48.659059 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:48.659147 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 10:29:48.659194 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260619 10:29:48.660283 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:48.660210 30359 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } }
I20260619 10:29:48.660331 30359 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:48.660351 30359 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd20bd8f93bd4dcd88dd74d1b00fc019, State: Initialized, Role: FOLLOWER
I20260619 10:29:48.660463 30359 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } }
I20260619 10:29:48.660529 30359 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:48.660550 30359 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:48.660595 30359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:48.661093 30359 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } }
I20260619 10:29:48.661180 30359 leader_election.cc:302] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd20bd8f93bd4dcd88dd74d1b00fc019; no voters:
I20260619 10:29:48.661335 30359 leader_election.cc:288] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:48.661358 30367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:48.661610 30359 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:48.661840 30367 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [term 1 LEADER]: Becoming Leader. State: Replica: bd20bd8f93bd4dcd88dd74d1b00fc019, State: Running, Role: LEADER
I20260619 10:29:48.661931 30367 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } }
I20260619 10:29:48.662262 30369 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } } }
I20260619 10:29:48.662353 30369 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:48.662256 30367 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd20bd8f93bd4dcd88dd74d1b00fc019. Latest consensus state: current_term: 1 leader_uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd20bd8f93bd4dcd88dd74d1b00fc019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } } }
I20260619 10:29:48.662631 30367 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:48.662734 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:48.661695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40981 (local address 127.0.0.1:45495)
0619 10:29:48.661726 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:48.661728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:48.661763 (+ 35us) server_negotiation.cc:407] Connection header received
0619 10:29:48.661865 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:48.661865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:48.661871 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:48.661915 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:48.662521 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:48.662522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:48.662522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:48.662526 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:48.662538 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:48.662542 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:48.662569 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:48.662670 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 10:29:48.662682 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:29:48.663332 30384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:29:48.663379 30384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:29:48.663522 30378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:48.663681 30378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:48.664911 30378 catalog_manager.cc:1347] Generated new cluster ID: 12f711c08b4f43ecbfa00c50759545da
I20260619 10:29:48.664958 30378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:48.669878 30378 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:48.670009 30378 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:48.673631 30378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd20bd8f93bd4dcd88dd74d1b00fc019: Generated new TSK 0
I20260619 10:29:48.673802 30378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:48.677333 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 30291
2026-06-19T10:29:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:48 dist-test-slave-hw7q krb5kdc[30390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:48 dist-test-slave-hw7q krb5kdc[30390](info): set up 2 sockets
Jun 19 10:29:48 dist-test-slave-hw7q krb5kdc[30390](info): commencing operation
krb5kdc: starting...
W20260619 10:29:49.629544 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:49 dist-test-slave-hw7q krb5kdc[30390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:49.734138 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:49.810245 30403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:49.810437 30403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:49.810458 30403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:49.811547 30403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:49.811581 30403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:49.811594 30403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:49.811605 30403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:49.811615 30403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:49.812971 30403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:49.813149 30403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:49.813325 30403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:49.817663 30411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:49.817677 30409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:49.817746 30408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:49.817916 30403 server_base.cc:1034] running on GCE node
I20260619 10:29:49.818127 30403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:49.818395 30403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:49.819531 30403 hybrid_clock.cc:648] HybridClock initialized: now 1781864989819513 us; error 33 us; skew 500 ppm
Jun 19 10:29:49 dist-test-slave-hw7q krb5kdc[30390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:49.821744 30403 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:49.822129 30403 webserver.cc:460] Webserver started at http://127.0.0.1:34501/ using document root <none> and password file <none>
I20260619 10:29:49.822336 30403 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:49.822383 30403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:49.822499 30403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:49.823448 30403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "ea04d82fffb542709c20c7369f6937ab"
format_stamp: "Formatted at 2026-06-19 10:29:49 on dist-test-slave-hw7q"
server_key: "720813420f219fe4ea8d7a4b5489d68d"
server_key_iv: "1f46c87ac8118f0e715d5308dcf64803"
server_key_version: "encryptionkey@0"
I20260619 10:29:49.823745 30403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "ea04d82fffb542709c20c7369f6937ab"
format_stamp: "Formatted at 2026-06-19 10:29:49 on dist-test-slave-hw7q"
server_key: "720813420f219fe4ea8d7a4b5489d68d"
server_key_iv: "1f46c87ac8118f0e715d5308dcf64803"
server_key_version: "encryptionkey@0"
I20260619 10:29:49.824865 30403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:29:49.825479 30418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:49.825636 30403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:49.825719 30403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "ea04d82fffb542709c20c7369f6937ab"
format_stamp: "Formatted at 2026-06-19 10:29:49 on dist-test-slave-hw7q"
server_key: "720813420f219fe4ea8d7a4b5489d68d"
server_key_iv: "1f46c87ac8118f0e715d5308dcf64803"
server_key_version: "encryptionkey@0"
I20260619 10:29:49.825776 30403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:49.843582 30403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:49.845109 30403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:49.845232 30403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:49.848839 30403 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33709
I20260619 10:29:49.848874 30470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33709 every 8 connection(s)
I20260619 10:29:49.849164 30403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:49.850147 30471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:49.852330 30471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab: Bootstrap starting.
I20260619 10:29:49.852861 30471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:49.853111 30471 log.cc:824] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:49.853698 30471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab: No bootstrap required, opened a new log
I20260619 10:29:49.855240 30471 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea04d82fffb542709c20c7369f6937ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } }
I20260619 10:29:49.855369 30471 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:49.855388 30471 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea04d82fffb542709c20c7369f6937ab, State: Initialized, Role: FOLLOWER
I20260619 10:29:49.855525 30471 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea04d82fffb542709c20c7369f6937ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } }
I20260619 10:29:49.855584 30471 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:49.855605 30471 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:49.855655 30471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:49.856128 30471 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea04d82fffb542709c20c7369f6937ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } }
I20260619 10:29:49.856210 30471 leader_election.cc:302] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea04d82fffb542709c20c7369f6937ab; no voters:
I20260619 10:29:49.856357 30471 leader_election.cc:288] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:49.856397 30474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:49.856531 30474 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [term 1 LEADER]: Becoming Leader. State: Replica: ea04d82fffb542709c20c7369f6937ab, State: Running, Role: LEADER
I20260619 10:29:49.856580 30471 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:49.856623 30474 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea04d82fffb542709c20c7369f6937ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } }
I20260619 10:29:49.856946 30476 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea04d82fffb542709c20c7369f6937ab. Latest consensus state: current_term: 1 leader_uuid: "ea04d82fffb542709c20c7369f6937ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea04d82fffb542709c20c7369f6937ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } } }
I20260619 10:29:49.857024 30476 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:49.857067 30475 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea04d82fffb542709c20c7369f6937ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea04d82fffb542709c20c7369f6937ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } } }
I20260619 10:29:49.857126 30475 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:49.857383 30481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:49.857857 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 30403
I20260619 10:29:49.857916 30481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:49.857981 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:49.858167 9577 external_mini_cluster.cc:1428] Setting key 58223968250bb5cec0a750617ea3fca7
I20260619 10:29:49.859367 30481 catalog_manager.cc:1347] Generated new cluster ID: 42e17acf57bb4735ae48d9fd3a657d0e
I20260619 10:29:49.859408 30481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:49.865326 30481 catalog_manager.cc:1370] Generated new certificate authority record
Jun 19 10:29:49 dist-test-slave-hw7q krb5kdc[30390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 10:29:49.865985 30481 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:49.869468 30493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:49.858607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:38503 (local address 127.0.0.1:33709)
0619 10:29:49.858758 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:49.858767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:49.858971 (+ 204us) server_negotiation.cc:407] Connection header received
0619 10:29:49.859114 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:49.859121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:49.859176 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:49.860068 (+ 892us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:49.860169 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:49.861584 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:49.862700 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:49.863348 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:49.863766 (+ 418us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:49.866147 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:49.866150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:49.866151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:49.866161 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:49.867960 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:49.868380 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:49.868382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:49.868383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:49.868472 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:49.868680 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:49.868681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:49.868681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:49.868920 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:49.869056 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:49.869165 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 10:29:49.869255 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260619 10:29:49.870098 30481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea04d82fffb542709c20c7369f6937ab: Generated new TSK 0
I20260619 10:29:49.870249 30481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:49.870345 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:49.877553 30493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:49.871540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:57295 (local address 127.0.0.1:33709)
0619 10:29:49.871573 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:49.871574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:49.871667 (+ 93us) server_negotiation.cc:407] Connection header received
0619 10:29:49.871727 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:49.871728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:49.871735 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:49.871776 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:49.872323 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:49.872827 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:49.873877 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:49.874130 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:49.874191 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:49.875395 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:49.875396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:49.875396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:49.875401 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:49.876518 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:49.876906 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:49.876907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:49.876907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:49.876961 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:49.877168 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:49.877168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:49.877168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:49.877255 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:49.877332 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:49.877448 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 10:29:49.877497 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:29:49.879297 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 30403
W20260619 10:29:49.882706 30388 connection.cc:535] client connection to 127.0.0.1:33709 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33709): Connection reset by peer (error 104)
2026-06-19T10:29:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:49 dist-test-slave-hw7q krb5kdc[30502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:49 dist-test-slave-hw7q krb5kdc[30502](info): set up 2 sockets
Jun 19 10:29:49 dist-test-slave-hw7q krb5kdc[30502](info): commencing operation
krb5kdc: starting...
W20260619 10:29:50.829439 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:50 dist-test-slave-hw7q krb5kdc[30502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:50.935732 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:51.016312 30515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:51.016566 30515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:51.016595 30515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:51.018208 30515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:51.018256 30515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:51.018275 30515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:51.018297 30515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:51.018313 30515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:51.019912 30515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:51.020092 30515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:51.020265 30515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:51.024580 30520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:51.024597 30523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:51.024567 30521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:51.024966 30515 server_base.cc:1034] running on GCE node
I20260619 10:29:51.025173 30515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:51.025427 30515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:51.026554 30515 hybrid_clock.cc:648] HybridClock initialized: now 1781864991026544 us; error 26 us; skew 500 ppm
Jun 19 10:29:51 dist-test-slave-hw7q krb5kdc[30502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:51.033558 30515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:51.034029 30515 webserver.cc:460] Webserver started at http://127.0.0.1:34669/ using document root <none> and password file <none>
I20260619 10:29:51.034250 30515 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:51.034297 30515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:51.034426 30515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:51.035481 30515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "b05aafa792734a6eba203df611b2c9ee"
format_stamp: "Formatted at 2026-06-19 10:29:51 on dist-test-slave-hw7q"
server_key: "592be30732e04de62370fef7f5e5d092"
server_key_iv: "418136a403e185dbc0c580fe215a7e72"
server_key_version: "encryptionkey@0"
I20260619 10:29:51.035806 30515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "b05aafa792734a6eba203df611b2c9ee"
format_stamp: "Formatted at 2026-06-19 10:29:51 on dist-test-slave-hw7q"
server_key: "592be30732e04de62370fef7f5e5d092"
server_key_iv: "418136a403e185dbc0c580fe215a7e72"
server_key_version: "encryptionkey@0"
I20260619 10:29:51.036978 30515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:29:51.037657 30530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:51.037846 30515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:51.037927 30515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "b05aafa792734a6eba203df611b2c9ee"
format_stamp: "Formatted at 2026-06-19 10:29:51 on dist-test-slave-hw7q"
server_key: "592be30732e04de62370fef7f5e5d092"
server_key_iv: "418136a403e185dbc0c580fe215a7e72"
server_key_version: "encryptionkey@0"
I20260619 10:29:51.037989 30515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:51.045992 30515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:51.047901 30515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:51.048025 30515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:51.051599 30515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42277
I20260619 10:29:51.051628 30582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42277 every 8 connection(s)
I20260619 10:29:51.051923 30515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:51.052788 30583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:51.054924 30583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee: Bootstrap starting.
I20260619 10:29:51.055423 30583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:51.055660 30583 log.cc:824] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:51.056255 30583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee: No bootstrap required, opened a new log
I20260619 10:29:51.057755 30583 raft_consensus.cc:348] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05aafa792734a6eba203df611b2c9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260619 10:29:51.057916 30583 raft_consensus.cc:374] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:51.057951 30583 raft_consensus.cc:729] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b05aafa792734a6eba203df611b2c9ee, State: Initialized, Role: FOLLOWER
I20260619 10:29:51.058084 30583 consensus_queue.cc:260] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05aafa792734a6eba203df611b2c9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260619 10:29:51.058138 30583 raft_consensus.cc:388] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:51.058171 30583 raft_consensus.cc:482] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:51.058215 30583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:51.058667 30583 raft_consensus.cc:504] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05aafa792734a6eba203df611b2c9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260619 10:29:51.058749 30583 leader_election.cc:302] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b05aafa792734a6eba203df611b2c9ee; no voters:
I20260619 10:29:51.058887 30583 leader_election.cc:288] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:51.058923 30586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:51.059059 30586 raft_consensus.cc:686] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [term 1 LEADER]: Becoming Leader. State: Replica: b05aafa792734a6eba203df611b2c9ee, State: Running, Role: LEADER
I20260619 10:29:51.059098 30583 sys_catalog.cc:564] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:51.059144 30586 consensus_queue.cc:237] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05aafa792734a6eba203df611b2c9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260619 10:29:51.059490 30588 sys_catalog.cc:455] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader b05aafa792734a6eba203df611b2c9ee. Latest consensus state: current_term: 1 leader_uuid: "b05aafa792734a6eba203df611b2c9ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05aafa792734a6eba203df611b2c9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } } }
I20260619 10:29:51.059566 30588 sys_catalog.cc:458] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:51.059684 30587 sys_catalog.cc:455] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b05aafa792734a6eba203df611b2c9ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05aafa792734a6eba203df611b2c9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } } }
I20260619 10:29:51.059762 30587 sys_catalog.cc:458] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:51.059821 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 30515
I20260619 10:29:51.059919 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:51.059957 30594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:51.060108 9577 external_mini_cluster.cc:1428] Setting key 7301c92d18ca67cc095ad4dddfcffab8
I20260619 10:29:51.060436 30594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:51.062013 30594 catalog_manager.cc:1347] Generated new cluster ID: d687c74866f6409ea3ff8e24a7b6d6ff
I20260619 10:29:51.062062 30594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:51.066242 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:51.060679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:41893 (local address 127.0.0.1:42277)
0619 10:29:51.061295 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:51.061302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:51.061324 (+ 22us) server_negotiation.cc:407] Connection header received
0619 10:29:51.061408 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:51.061413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:51.061453 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:51.061556 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:51.061771 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:51.063123 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:51.064209 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:51.064812 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:51.064914 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:51.065550 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:51.065555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:51.065555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:51.065756 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:51.065823 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:51.065840 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:51.065917 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:51.065995 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 10:29:51.066024 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":90,"threads_started":1}
I20260619 10:29:51.067382 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:51.072986 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:51.068798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:48367 (local address 127.0.0.1:42277)
0619 10:29:51.068828 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:51.068830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:51.069934 (+ 1104us) server_negotiation.cc:407] Connection header received
0619 10:29:51.069995 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:51.069996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:51.070006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:51.070057 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:51.070151 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:51.070871 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:51.071785 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:51.072123 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:51.072193 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:51.072761 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:51.072763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:51.072763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:51.072767 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:51.072780 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:51.072785 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:51.072841 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:51.072901 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 10:29:51.072913 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:29:51.081321 30594 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:51.082000 30594 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:51.084846 30594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b05aafa792734a6eba203df611b2c9ee: Generated new TSK 0
I20260619 10:29:51.085006 30594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:51.092219 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 30515
W20260619 10:29:51.095719 30500 connection.cc:535] client connection to 127.0.0.1:42277 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42277): Connection reset by peer (error 104)
2026-06-19T10:29:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:51 dist-test-slave-hw7q krb5kdc[30614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:51 dist-test-slave-hw7q krb5kdc[30614](info): set up 2 sockets
Jun 19 10:29:51 dist-test-slave-hw7q krb5kdc[30614](info): commencing operation
krb5kdc: starting...
W20260619 10:29:52.029740 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:52 dist-test-slave-hw7q krb5kdc[30614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:52.137877 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:52.215571 30627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:52.215786 30627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:52.215808 30627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:52.216918 30627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:52.216953 30627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:52.216966 30627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:52.216977 30627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:52.216988 30627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:52.218418 30627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:52.218616 30627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:52.218787 30627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:52.223187 30632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:52.223297 30627 server_base.cc:1034] running on GCE node
W20260619 10:29:52.223192 30633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:52.223372 30635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:52.223611 30627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:52.223866 30627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:52.225000 30627 hybrid_clock.cc:648] HybridClock initialized: now 1781864992224988 us; error 27 us; skew 500 ppm
Jun 19 10:29:52 dist-test-slave-hw7q krb5kdc[30614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:52.227341 30627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:52.227731 30627 webserver.cc:460] Webserver started at http://127.0.0.1:40299/ using document root <none> and password file <none>
I20260619 10:29:52.227921 30627 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:52.227962 30627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:52.228080 30627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:52.229043 30627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "04c90ecd2ef6404d960b05d85a559e18"
format_stamp: "Formatted at 2026-06-19 10:29:52 on dist-test-slave-hw7q"
server_key: "fcbb78c513f5f4974d376b2812c8bd21"
server_key_iv: "8c65e3cd0a945561b1b2eeed9dd98111"
server_key_version: "encryptionkey@0"
I20260619 10:29:52.229341 30627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "04c90ecd2ef6404d960b05d85a559e18"
format_stamp: "Formatted at 2026-06-19 10:29:52 on dist-test-slave-hw7q"
server_key: "fcbb78c513f5f4974d376b2812c8bd21"
server_key_iv: "8c65e3cd0a945561b1b2eeed9dd98111"
server_key_version: "encryptionkey@0"
I20260619 10:29:52.230592 30627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:29:52.231216 30642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:52.231359 30627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:52.231436 30627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "04c90ecd2ef6404d960b05d85a559e18"
format_stamp: "Formatted at 2026-06-19 10:29:52 on dist-test-slave-hw7q"
server_key: "fcbb78c513f5f4974d376b2812c8bd21"
server_key_iv: "8c65e3cd0a945561b1b2eeed9dd98111"
server_key_version: "encryptionkey@0"
I20260619 10:29:52.231498 30627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:52.234570 30627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:52.236156 30627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:52.236263 30627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:52.239779 30627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42377
I20260619 10:29:52.239807 30694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42377 every 8 connection(s)
I20260619 10:29:52.240084 30627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:52.240918 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:52.241729 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 30627
I20260619 10:29:52.241870 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:52.242082 9577 external_mini_cluster.cc:1428] Setting key d69152ef39dfdebd671d410238e2970b
I20260619 10:29:52.243381 30695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18: Bootstrap starting.
I20260619 10:29:52.244011 30695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:52.244292 30695 log.cc:824] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:52.244323 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:52.242543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:52603 (local address 127.0.0.1:42377)
0619 10:29:52.242695 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:52.242702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:52.242725 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:29:52.243209 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:52.243216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:52.243239 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:52.243346 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:52.243928 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:52.243932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:52.243932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:52.243935 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:52.243970 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:52.243979 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:52.244040 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:52.244110 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:29:52.244151 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260619 10:29:52.244879 30695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18: No bootstrap required, opened a new log
I20260619 10:29:52.245255 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:52.246805 30695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c90ecd2ef6404d960b05d85a559e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42377 } }
I20260619 10:29:52.246955 30695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:52.247038 30695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c90ecd2ef6404d960b05d85a559e18, State: Initialized, Role: FOLLOWER
I20260619 10:29:52.247169 30695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c90ecd2ef6404d960b05d85a559e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42377 } }
I20260619 10:29:52.247264 30695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:52.247295 30695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:52.247342 30695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:52.247421 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:52.246522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:36599 (local address 127.0.0.1:42377)
0619 10:29:52.246562 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:52.246563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:52.246626 (+ 63us) server_negotiation.cc:407] Connection header received
0619 10:29:52.246650 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:52.246651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:52.246654 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:52.246724 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:52.247205 (+ 481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:52.247206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:52.247206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:52.247210 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:52.247222 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:52.247226 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:52.247265 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:52.247354 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 10:29:52.247365 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 10:29:52.247941 30695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c90ecd2ef6404d960b05d85a559e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42377 } }
I20260619 10:29:52.248026 30695 leader_election.cc:302] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04c90ecd2ef6404d960b05d85a559e18; no voters:
I20260619 10:29:52.248171 30695 leader_election.cc:288] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:52.248204 30705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:52.248327 30705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [term 1 LEADER]: Becoming Leader. State: Replica: 04c90ecd2ef6404d960b05d85a559e18, State: Running, Role: LEADER
I20260619 10:29:52.248394 30695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:52.248448 30705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c90ecd2ef6404d960b05d85a559e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42377 } }
I20260619 10:29:52.248835 30707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c90ecd2ef6404d960b05d85a559e18. Latest consensus state: current_term: 1 leader_uuid: "04c90ecd2ef6404d960b05d85a559e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c90ecd2ef6404d960b05d85a559e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42377 } } }
I20260619 10:29:52.248941 30707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:52.248826 30706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04c90ecd2ef6404d960b05d85a559e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c90ecd2ef6404d960b05d85a559e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42377 } } }
I20260619 10:29:52.249191 30706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:52.249220 30714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:52.249733 30714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:52.251077 30714 catalog_manager.cc:1347] Generated new cluster ID: 16b0d2ea2b41430a8cbda71033b9a295
I20260619 10:29:52.251127 30714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:52.256928 30714 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:52.257095 30714 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:52.266031 30714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04c90ecd2ef6404d960b05d85a559e18: Generated new TSK 0
I20260619 10:29:52.266232 30714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:52.267903 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 30627
2026-06-19T10:29:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1175 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:52 dist-test-slave-hw7q krb5kdc[30726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:52 dist-test-slave-hw7q krb5kdc[30726](info): set up 2 sockets
Jun 19 10:29:52 dist-test-slave-hw7q krb5kdc[30726](info): commencing operation
krb5kdc: starting...
W20260619 10:29:53.205479 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:53 dist-test-slave-hw7q krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:53.312525 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:53.389364 30739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:53.389567 30739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:53.389588 30739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:53.390731 30739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:53.390776 30739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:53.390789 30739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:53.390805 30739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:53.390815 30739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:53.392251 30739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:53.392434 30739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:53.392597 30739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:53.396996 30747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:53.397011 30744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:53.397084 30745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:53.397220 30739 server_base.cc:1034] running on GCE node
I20260619 10:29:53.397466 30739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:53.397701 30739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:53.398845 30739 hybrid_clock.cc:648] HybridClock initialized: now 1781864993398828 us; error 34 us; skew 500 ppm
Jun 19 10:29:53 dist-test-slave-hw7q krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:53.401052 30739 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:53.401448 30739 webserver.cc:460] Webserver started at http://127.0.0.1:34237/ using document root <none> and password file <none>
I20260619 10:29:53.401633 30739 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:53.401677 30739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:53.401820 30739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:53.402807 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "6038a98be60f400681964b25f79ee031"
format_stamp: "Formatted at 2026-06-19 10:29:53 on dist-test-slave-hw7q"
server_key: "c7545f3c6446617636bb8f0d9c47532c"
server_key_iv: "dfd52705b642fd5c6a4d4a1d4135e366"
server_key_version: "encryptionkey@0"
I20260619 10:29:53.403120 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "6038a98be60f400681964b25f79ee031"
format_stamp: "Formatted at 2026-06-19 10:29:53 on dist-test-slave-hw7q"
server_key: "c7545f3c6446617636bb8f0d9c47532c"
server_key_iv: "dfd52705b642fd5c6a4d4a1d4135e366"
server_key_version: "encryptionkey@0"
I20260619 10:29:53.404291 30739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:29:53.404946 30754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:53.405134 30739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:53.405220 30739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "6038a98be60f400681964b25f79ee031"
format_stamp: "Formatted at 2026-06-19 10:29:53 on dist-test-slave-hw7q"
server_key: "c7545f3c6446617636bb8f0d9c47532c"
server_key_iv: "dfd52705b642fd5c6a4d4a1d4135e366"
server_key_version: "encryptionkey@0"
I20260619 10:29:53.405282 30739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:53.425168 30739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:53.427030 30739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:53.427162 30739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:53.430876 30739 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41741
I20260619 10:29:53.430927 30806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41741 every 8 connection(s)
I20260619 10:29:53.431247 30739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:53.432190 30807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:53.434511 30807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031: Bootstrap starting.
I20260619 10:29:53.435051 30807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:53.435293 30807 log.cc:824] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:53.435886 30807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031: No bootstrap required, opened a new log
I20260619 10:29:53.436502 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 30739
I20260619 10:29:53.436604 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:53.436784 9577 external_mini_cluster.cc:1428] Setting key ed7e75164e6c4b5c1c91a527b66d7906
I20260619 10:29:53.437759 30807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038a98be60f400681964b25f79ee031" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41741 } }
I20260619 10:29:53.437944 30807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:53.438020 30807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6038a98be60f400681964b25f79ee031, State: Initialized, Role: FOLLOWER
I20260619 10:29:53.438148 30807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038a98be60f400681964b25f79ee031" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41741 } }
I20260619 10:29:53.438216 30807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:53.438237 30807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:53.438280 30807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:53.439242 30807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038a98be60f400681964b25f79ee031" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41741 } }
I20260619 10:29:53.439344 30807 leader_election.cc:302] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6038a98be60f400681964b25f79ee031; no voters:
I20260619 10:29:53.439502 30807 leader_election.cc:288] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:53.439533 30812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:53.439663 30812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [term 1 LEADER]: Becoming Leader. State: Replica: 6038a98be60f400681964b25f79ee031, State: Running, Role: LEADER
I20260619 10:29:53.439718 30807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:53.439841 30812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038a98be60f400681964b25f79ee031" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41741 } }
I20260619 10:29:53.440229 30814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6038a98be60f400681964b25f79ee031" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038a98be60f400681964b25f79ee031" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41741 } } }
I20260619 10:29:53.440450 30814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:53.440379 30813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6038a98be60f400681964b25f79ee031. Latest consensus state: current_term: 1 leader_uuid: "6038a98be60f400681964b25f79ee031" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038a98be60f400681964b25f79ee031" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41741 } } }
I20260619 10:29:53.440804 30813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:53.440898 30821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:53.441416 30821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:53.443310 30821 catalog_manager.cc:1347] Generated new cluster ID: 1c112ba1f58a4922acd140e9f98d2a58
I20260619 10:29:53.443353 30821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 10:29:53 dist-test-slave-hw7q krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 10:29:53.448381 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:53.437283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:39571 (local address 127.0.0.1:41741)
0619 10:29:53.437501 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:53.437510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:53.437547 (+ 37us) server_negotiation.cc:407] Connection header received
0619 10:29:53.437718 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:53.437724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:53.437801 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:53.437945 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:53.438616 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:53.439999 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:53.441027 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:53.441599 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:53.441697 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:53.444438 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:53.444442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:53.444442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:53.444453 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:53.446320 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:53.446746 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:53.446748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:53.446749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:53.447281 (+ 532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:53.447517 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:53.447518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:53.447518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:53.447798 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:53.447946 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:53.448072 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 10:29:53.448165 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260619 10:29:53.449456 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:53.458261 30821 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:53.458890 30821 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:53.461369 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:53.454046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:37137 (local address 127.0.0.1:41741)
0619 10:29:53.454087 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:53.454089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:53.454171 (+ 82us) server_negotiation.cc:407] Connection header received
0619 10:29:53.454242 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:53.454242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:53.454251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:53.454299 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:53.454892 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:53.455624 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:53.456609 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:53.456920 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:53.456983 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:53.458269 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:53.458270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:53.458270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:53.458277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:53.460268 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:53.460654 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:53.460654 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:53.460655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:53.460713 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:53.460912 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:53.460913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:53.460913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:53.461016 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:53.461109 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:53.461230 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 10:29:53.461285 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":19}
I20260619 10:29:53.462201 30821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6038a98be60f400681964b25f79ee031: Generated new TSK 0
I20260619 10:29:53.462373 30821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:53.465637 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 30739
W20260619 10:29:53.469277 30724 connection.cc:535] client connection to 127.0.0.1:41741 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41741): Connection reset by peer (error 104)
2026-06-19T10:29:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:53 dist-test-slave-hw7q krb5kdc[30838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:53 dist-test-slave-hw7q krb5kdc[30838](info): set up 2 sockets
Jun 19 10:29:53 dist-test-slave-hw7q krb5kdc[30838](info): commencing operation
krb5kdc: starting...
W20260619 10:29:54.409765 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:54 dist-test-slave-hw7q krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:54.520820 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:54.598860 30851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:54.599066 30851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:54.599097 30851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:54.600241 30851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:54.600284 30851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:54.600304 30851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:54.600320 30851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:54.600335 30851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:54.601825 30851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:54.602056 30851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:54.602257 30851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:54.606689 30856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:54.606725 30859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:54.606688 30857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:54.606899 30851 server_base.cc:1034] running on GCE node
I20260619 10:29:54.607117 30851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:54.607395 30851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:54.608541 30851 hybrid_clock.cc:648] HybridClock initialized: now 1781864994608528 us; error 29 us; skew 500 ppm
Jun 19 10:29:54 dist-test-slave-hw7q krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:54.610870 30851 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:54.611294 30851 webserver.cc:460] Webserver started at http://127.0.0.1:37883/ using document root <none> and password file <none>
I20260619 10:29:54.611497 30851 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:54.611541 30851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:54.611662 30851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:54.612655 30851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "9473d506ffa949d780bc20d50dd0744b"
format_stamp: "Formatted at 2026-06-19 10:29:54 on dist-test-slave-hw7q"
server_key: "65a8fce194401e2fc237f216bb7fd9c5"
server_key_iv: "4043577e569fc1e837287f6e1d4931e0"
server_key_version: "encryptionkey@0"
I20260619 10:29:54.612967 30851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "9473d506ffa949d780bc20d50dd0744b"
format_stamp: "Formatted at 2026-06-19 10:29:54 on dist-test-slave-hw7q"
server_key: "65a8fce194401e2fc237f216bb7fd9c5"
server_key_iv: "4043577e569fc1e837287f6e1d4931e0"
server_key_version: "encryptionkey@0"
I20260619 10:29:54.614190 30851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 10:29:54.614811 30866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:54.614928 30851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:29:54.615001 30851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "9473d506ffa949d780bc20d50dd0744b"
format_stamp: "Formatted at 2026-06-19 10:29:54 on dist-test-slave-hw7q"
server_key: "65a8fce194401e2fc237f216bb7fd9c5"
server_key_iv: "4043577e569fc1e837287f6e1d4931e0"
server_key_version: "encryptionkey@0"
I20260619 10:29:54.615063 30851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:54.624756 30851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:54.626667 30851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:54.626776 30851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:54.630339 30851 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34083
I20260619 10:29:54.630369 30918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34083 every 8 connection(s)
I20260619 10:29:54.630656 30851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:54.631397 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:54.633471 30919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b: Bootstrap starting.
I20260619 10:29:54.634002 30919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:54.634243 30919 log.cc:824] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:54.634840 30919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b: No bootstrap required, opened a new log
I20260619 10:29:54.634934 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 30851
I20260619 10:29:54.635046 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:54.635237 9577 external_mini_cluster.cc:1428] Setting key 4f82d6cbbe6a3405e81dd83c9155f3ef
I20260619 10:29:54.636693 30919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9473d506ffa949d780bc20d50dd0744b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260619 10:29:54.636826 30919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:54.636857 30919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9473d506ffa949d780bc20d50dd0744b, State: Initialized, Role: FOLLOWER
I20260619 10:29:54.637004 30919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9473d506ffa949d780bc20d50dd0744b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260619 10:29:54.637087 30919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:54.637115 30919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:54.637168 30919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:54.637660 30919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9473d506ffa949d780bc20d50dd0744b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260619 10:29:54.637739 30919 leader_election.cc:302] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9473d506ffa949d780bc20d50dd0744b; no voters:
I20260619 10:29:54.637902 30919 leader_election.cc:288] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:54.637936 30924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:54.638077 30924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [term 1 LEADER]: Becoming Leader. State: Replica: 9473d506ffa949d780bc20d50dd0744b, State: Running, Role: LEADER
I20260619 10:29:54.638141 30919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:54.638202 30924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9473d506ffa949d780bc20d50dd0744b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260619 10:29:54.638537 30924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9473d506ffa949d780bc20d50dd0744b. Latest consensus state: current_term: 1 leader_uuid: "9473d506ffa949d780bc20d50dd0744b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9473d506ffa949d780bc20d50dd0744b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } } }
I20260619 10:29:54.638737 30924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:54.638675 30926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9473d506ffa949d780bc20d50dd0744b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9473d506ffa949d780bc20d50dd0744b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } } }
I20260619 10:29:54.638921 30926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:54.638917 30933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:54.639513 30933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:54.640990 30933 catalog_manager.cc:1347] Generated new cluster ID: 9f25d2bb180c4fa488d3c8028fddf597
I20260619 10:29:54.641075 30933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:54.641557 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:54.635679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:43025 (local address 127.0.0.1:34083)
0619 10:29:54.635843 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:54.635851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:54.635875 (+ 24us) server_negotiation.cc:407] Connection header received
0619 10:29:54.636667 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:54.636679 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:54.636725 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:54.636843 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:54.637062 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:54.638450 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:54.639542 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:54.640102 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:54.640217 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:54.640880 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:54.640884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:54.640884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:54.641083 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:54.641122 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:54.641137 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:54.641233 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:54.641339 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 10:29:54.641362 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260619 10:29:54.642377 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:54.646984 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:54.643738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:50197 (local address 127.0.0.1:34083)
0619 10:29:54.643771 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:54.643772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:54.643916 (+ 144us) server_negotiation.cc:407] Connection header received
0619 10:29:54.643953 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:54.643954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:54.643964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:54.644017 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:54.644132 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:54.644840 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:54.645765 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:54.646116 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:54.646194 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:54.646753 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:54.646754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:54.646755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:54.646759 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:54.646773 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:54.646777 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:54.646839 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:54.646899 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 10:29:54.646910 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:29:54.651585 30933 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:54.652046 30933 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:54.654538 30933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9473d506ffa949d780bc20d50dd0744b: Generated new TSK 0
I20260619 10:29:54.654683 30933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:54.660538 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 30851
W20260619 10:29:54.663872 30836 connection.cc:535] client connection to 127.0.0.1:34083 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34083): Connection reset by peer (error 104)
2026-06-19T10:29:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:54 dist-test-slave-hw7q krb5kdc[30950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:54 dist-test-slave-hw7q krb5kdc[30950](info): set up 2 sockets
Jun 19 10:29:54 dist-test-slave-hw7q krb5kdc[30950](info): commencing operation
krb5kdc: starting...
W20260619 10:29:55.617496 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:55 dist-test-slave-hw7q krb5kdc[30950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 10:29:55.725978 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:55.801959 30963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:55.802161 30963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:55.802192 30963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:55.803282 30963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:55.803320 30963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:55.803339 30963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:55.803354 30963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:55.803370 30963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:55.804768 30963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:55.804991 30963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:55.805190 30963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:55.809593 30969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:55.809612 30968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:55.809653 30971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:55.809942 30963 server_base.cc:1034] running on GCE node
I20260619 10:29:55.810140 30963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:55.810375 30963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:55.811512 30963 hybrid_clock.cc:648] HybridClock initialized: now 1781864995811499 us; error 29 us; skew 500 ppm
Jun 19 10:29:55 dist-test-slave-hw7q krb5kdc[30950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:55.813798 30963 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 10:29:55.814206 30963 webserver.cc:460] Webserver started at http://127.0.0.1:39841/ using document root <none> and password file <none>
I20260619 10:29:55.814399 30963 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:55.814440 30963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:55.814549 30963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:55.815518 30963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "a02cf92b599b40cd8d0fc5d870617665"
format_stamp: "Formatted at 2026-06-19 10:29:55 on dist-test-slave-hw7q"
server_key: "fc27e43cdffe886ff37232e16aaca9eb"
server_key_iv: "0f10b7ea85a435a8cde55c17755d552e"
server_key_version: "encryptionkey@0"
I20260619 10:29:55.815827 30963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "a02cf92b599b40cd8d0fc5d870617665"
format_stamp: "Formatted at 2026-06-19 10:29:55 on dist-test-slave-hw7q"
server_key: "fc27e43cdffe886ff37232e16aaca9eb"
server_key_iv: "0f10b7ea85a435a8cde55c17755d552e"
server_key_version: "encryptionkey@0"
I20260619 10:29:55.816972 30963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:29:55.817577 30978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:55.817731 30963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:55.817827 30963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "a02cf92b599b40cd8d0fc5d870617665"
format_stamp: "Formatted at 2026-06-19 10:29:55 on dist-test-slave-hw7q"
server_key: "fc27e43cdffe886ff37232e16aaca9eb"
server_key_iv: "0f10b7ea85a435a8cde55c17755d552e"
server_key_version: "encryptionkey@0"
I20260619 10:29:55.817885 30963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:55.820971 30963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:55.822609 30963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:55.822729 30963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:55.826162 30963 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40421
I20260619 10:29:55.826200 31030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40421 every 8 connection(s)
I20260619 10:29:55.826467 30963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:55.827309 31031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:55.829370 31031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665: Bootstrap starting.
I20260619 10:29:55.829583 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 30963
I20260619 10:29:55.829691 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:55.829901 9577 external_mini_cluster.cc:1428] Setting key d60dce16f5d4a245d95818cb408683c1
I20260619 10:29:55.829983 31031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:55.830296 31031 log.cc:824] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:55.830932 31031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665: No bootstrap required, opened a new log
I20260619 10:29:55.832053 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:55.830333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:52483 (local address 127.0.0.1:40421)
0619 10:29:55.830495 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:55.830503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:55.830535 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:29:55.830974 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:55.830983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:55.831016 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:55.831136 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:55.831685 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:55.831688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:55.831688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:55.831692 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:55.831723 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:55.831733 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:55.831795 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:55.831851 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 10:29:55.831883 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260619 10:29:55.832877 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:55.833108 31031 raft_consensus.cc:348] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02cf92b599b40cd8d0fc5d870617665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } }
I20260619 10:29:55.833241 31031 raft_consensus.cc:374] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:55.833263 31031 raft_consensus.cc:729] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02cf92b599b40cd8d0fc5d870617665, State: Initialized, Role: FOLLOWER
I20260619 10:29:55.833372 31031 consensus_queue.cc:260] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02cf92b599b40cd8d0fc5d870617665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } }
I20260619 10:29:55.833439 31031 raft_consensus.cc:388] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:55.833461 31031 raft_consensus.cc:482] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:55.833503 31031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:55.834075 31031 raft_consensus.cc:504] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02cf92b599b40cd8d0fc5d870617665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } }
I20260619 10:29:55.834188 31031 leader_election.cc:302] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a02cf92b599b40cd8d0fc5d870617665; no voters:
I20260619 10:29:55.834342 31031 leader_election.cc:288] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:55.834374 31041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:55.834510 31041 raft_consensus.cc:686] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [term 1 LEADER]: Becoming Leader. State: Replica: a02cf92b599b40cd8d0fc5d870617665, State: Running, Role: LEADER
I20260619 10:29:55.834537 31031 sys_catalog.cc:564] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:55.835470 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:55.834213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:40227 (local address 127.0.0.1:40421)
0619 10:29:55.834244 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:55.834245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:55.834679 (+ 434us) server_negotiation.cc:407] Connection header received
0619 10:29:55.834722 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:55.834723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:55.834727 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:55.834783 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:55.835293 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:55.835294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:55.835295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:55.835299 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:55.835310 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:55.835313 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:55.835372 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:55.835405 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 10:29:55.835416 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:29:55.835906 31041 consensus_queue.cc:237] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02cf92b599b40cd8d0fc5d870617665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } }
I20260619 10:29:55.836222 31041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a02cf92b599b40cd8d0fc5d870617665. Latest consensus state: current_term: 1 leader_uuid: "a02cf92b599b40cd8d0fc5d870617665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02cf92b599b40cd8d0fc5d870617665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } } }
I20260619 10:29:55.836335 31041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:55.836405 31043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a02cf92b599b40cd8d0fc5d870617665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02cf92b599b40cd8d0fc5d870617665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } } }
I20260619 10:29:55.836627 31043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665 [sys.catalog]: This master's current role is: LEADER
W20260619 10:29:55.837011 31055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:29:55.837050 31055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:29:55.837098 31056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:55.837250 31056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:55.838459 31056 catalog_manager.cc:1347] Generated new cluster ID: e86d900074804c6ab14fa846cec6ede4
I20260619 10:29:55.838506 31056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:55.845283 31056 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:55.845419 31056 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:55.851965 31056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a02cf92b599b40cd8d0fc5d870617665: Generated new TSK 0
I20260619 10:29:55.852093 31056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:55.859879 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 30963
2026-06-19T10:29:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:55 dist-test-slave-hw7q krb5kdc[31062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:55 dist-test-slave-hw7q krb5kdc[31062](info): set up 2 sockets
Jun 19 10:29:55 dist-test-slave-hw7q krb5kdc[31062](info): commencing operation
krb5kdc: starting...
W20260619 10:29:56.813567 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:56 dist-test-slave-hw7q krb5kdc[31062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:29:56.920334 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40069
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:56.996918 31075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:56.997116 31075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:56.997136 31075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:56.998311 31075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:56.998359 31075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:56.998378 31075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:56.998391 31075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:56.998407 31075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:56.999853 31075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:40069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:57.000085 31075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:57.000283 31075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:57.004669 31083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:57.004698 31081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:57.004731 31080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:57.005087 31075 server_base.cc:1034] running on GCE node
I20260619 10:29:57.006307 31075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:57.006628 31075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:57.007812 31075 hybrid_clock.cc:648] HybridClock initialized: now 1781864997007784 us; error 39 us; skew 500 ppm
Jun 19 10:29:57 dist-test-slave-hw7q krb5kdc[31062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864997, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:57.013906 31075 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:29:57.014333 31075 webserver.cc:460] Webserver started at http://127.9.90.126:35867/ using document root <none> and password file <none>
I20260619 10:29:57.014513 31075 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:57.014550 31075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:57.014647 31075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:57.015621 31075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "45161177f188437681e066c248bebfae"
format_stamp: "Formatted at 2026-06-19 10:29:57 on dist-test-slave-hw7q"
server_key: "7c1da1435311aff10acbfa43394d5fe9"
server_key_iv: "899c19c36a109a0524d5c1f10fdbd230"
server_key_version: "encryptionkey@0"
I20260619 10:29:57.015911 31075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "45161177f188437681e066c248bebfae"
format_stamp: "Formatted at 2026-06-19 10:29:57 on dist-test-slave-hw7q"
server_key: "7c1da1435311aff10acbfa43394d5fe9"
server_key_iv: "899c19c36a109a0524d5c1f10fdbd230"
server_key_version: "encryptionkey@0"
I20260619 10:29:57.017092 31075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:57.017757 31090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:57.017915 31075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:57.018002 31075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "45161177f188437681e066c248bebfae"
format_stamp: "Formatted at 2026-06-19 10:29:57 on dist-test-slave-hw7q"
server_key: "7c1da1435311aff10acbfa43394d5fe9"
server_key_iv: "899c19c36a109a0524d5c1f10fdbd230"
server_key_version: "encryptionkey@0"
I20260619 10:29:57.018069 31075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:57.037111 31075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:57.038924 31075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:57.039065 31075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:57.042794 31075 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:40069
I20260619 10:29:57.042882 31142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:40069 every 8 connection(s)
I20260619 10:29:57.043179 31075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:57.043972 31143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:57.044420 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 31075
I20260619 10:29:57.044507 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:57.044679 9577 external_mini_cluster.cc:1428] Setting key 56378b69793b85db20e1d069136775c3
I20260619 10:29:57.046949 31143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae: Bootstrap starting.
I20260619 10:29:57.047523 31143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:57.047777 31143 log.cc:824] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:57.048357 31143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae: No bootstrap required, opened a new log
I20260619 10:29:57.050166 31143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45161177f188437681e066c248bebfae" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40069 } }
I20260619 10:29:57.050298 31143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:57.050318 31143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45161177f188437681e066c248bebfae, State: Initialized, Role: FOLLOWER
I20260619 10:29:57.050424 31143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45161177f188437681e066c248bebfae" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40069 } }
I20260619 10:29:57.050484 31143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:57.050505 31143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:57.050558 31143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [term 0 FOLLOWER]: Advancing to term 1
Jun 19 10:29:57 dist-test-slave-hw7q krb5kdc[31062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:29:57.051204 31143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45161177f188437681e066c248bebfae" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40069 } }
I20260619 10:29:57.051296 31143 leader_election.cc:302] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45161177f188437681e066c248bebfae; no voters:
I20260619 10:29:57.051446 31143 leader_election.cc:288] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:57.051501 31148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:57.051641 31148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [term 1 LEADER]: Becoming Leader. State: Replica: 45161177f188437681e066c248bebfae, State: Running, Role: LEADER
I20260619 10:29:57.051699 31143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:57.051813 31148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45161177f188437681e066c248bebfae" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40069 } }
I20260619 10:29:57.052124 31148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45161177f188437681e066c248bebfae. Latest consensus state: current_term: 1 leader_uuid: "45161177f188437681e066c248bebfae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45161177f188437681e066c248bebfae" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40069 } } }
I20260619 10:29:57.052253 31148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:57.052162 31150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45161177f188437681e066c248bebfae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45161177f188437681e066c248bebfae" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40069 } } }
I20260619 10:29:57.052416 31150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:57.052618 31157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:57.053117 31157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:57.055328 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:57.045356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55487 (local address 127.9.90.126:40069)
0619 10:29:57.045556 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:57.045564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:57.045598 (+ 34us) server_negotiation.cc:407] Connection header received
0619 10:29:57.046134 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:57.046142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:57.046194 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:57.046317 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:57.047081 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:57.048026 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:57.048802 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:57.049221 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:57.049313 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:57.051824 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:57.051828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:57.051828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:57.051839 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:57.053716 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:57.054232 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:57.054235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:57.054235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:57.054337 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:57.054569 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:57.054570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:57.054570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:57.054812 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:57.054954 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:57.055074 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 10:29:57.055177 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260619 10:29:57.056129 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:57.057861 31157 catalog_manager.cc:1347] Generated new cluster ID: d1285ec678674ee88e33f797b5c2a309
I20260619 10:29:57.057910 31157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:57.062733 31157 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:57.063004 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:57.057348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52117 (local address 127.9.90.126:40069)
0619 10:29:57.057389 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:57.057390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:57.057477 (+ 87us) server_negotiation.cc:407] Connection header received
0619 10:29:57.057535 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:57.057536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:57.057545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:57.057585 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:57.058128 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:57.058820 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:57.059464 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:57.059687 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:57.059739 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:57.060777 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:57.060778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:57.060779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:29:57.060785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:57.061869 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:57.062204 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:57.062206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:57.062206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:57.062268 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:29:57.062495 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:29:57.062495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:29:57.062496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:29:57.062614 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:57.062709 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:57.062870 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 10:29:57.062926 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 10:29:57.063392 31157 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:57.070998 31157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45161177f188437681e066c248bebfae: Generated new TSK 0
I20260619 10:29:57.071149 31157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:57.073660 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 31075
W20260619 10:29:57.077116 31060 connection.cc:535] client connection to 127.9.90.126:40069 recv error: Network error: failed to read from TLS socket (remote: 127.9.90.126:40069): Connection reset by peer (error 104)
2026-06-19T10:29:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:57 dist-test-slave-hw7q krb5kdc[31174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:57 dist-test-slave-hw7q krb5kdc[31174](info): set up 2 sockets
Jun 19 10:29:57 dist-test-slave-hw7q krb5kdc[31174](info): commencing operation
krb5kdc: starting...
W20260619 10:29:58.021533 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:58 dist-test-slave-hw7q krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:29:58.130213 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45883
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:45883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:58.206576 31187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:58.206773 31187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:58.206805 31187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:58.207907 31187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:58.207950 31187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:58.207969 31187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:58.207985 31187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:58.208001 31187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:58.209451 31187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:45883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:58.209682 31187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:58.209893 31187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:58.214258 31193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:58.214282 31192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:58.214457 31187 server_base.cc:1034] running on GCE node
W20260619 10:29:58.214282 31195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:58.214731 31187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:58.214969 31187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:58.216110 31187 hybrid_clock.cc:648] HybridClock initialized: now 1781864998216085 us; error 42 us; skew 500 ppm
Jun 19 10:29:58 dist-test-slave-hw7q krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864998, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:58.218310 31187 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:29:58.218710 31187 webserver.cc:460] Webserver started at http://127.9.90.126:39899/ using document root <none> and password file <none>
I20260619 10:29:58.218900 31187 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:58.218941 31187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:58.219063 31187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:58.220036 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "73a41961308f4460a416d37f0e6fdec5"
format_stamp: "Formatted at 2026-06-19 10:29:58 on dist-test-slave-hw7q"
server_key: "1894115a5417cdc3dc3dace515a463f2"
server_key_iv: "602e0d150c41c94159780b340ab7c4f4"
server_key_version: "encryptionkey@0"
I20260619 10:29:58.220353 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "73a41961308f4460a416d37f0e6fdec5"
format_stamp: "Formatted at 2026-06-19 10:29:58 on dist-test-slave-hw7q"
server_key: "1894115a5417cdc3dc3dace515a463f2"
server_key_iv: "602e0d150c41c94159780b340ab7c4f4"
server_key_version: "encryptionkey@0"
I20260619 10:29:58.221515 31187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:58.222194 31202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:58.222342 31187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:58.222417 31187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "73a41961308f4460a416d37f0e6fdec5"
format_stamp: "Formatted at 2026-06-19 10:29:58 on dist-test-slave-hw7q"
server_key: "1894115a5417cdc3dc3dace515a463f2"
server_key_iv: "602e0d150c41c94159780b340ab7c4f4"
server_key_version: "encryptionkey@0"
I20260619 10:29:58.222483 31187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:58.231745 31187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:58.233242 31187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:58.233347 31187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:58.236840 31187 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:45883
I20260619 10:29:58.236865 31254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:45883 every 8 connection(s)
I20260619 10:29:58.237143 31187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:58.237924 31255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:58.239976 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5: Bootstrap starting.
I20260619 10:29:58.240477 31255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:58.240746 31255 log.cc:824] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:58.241475 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5: No bootstrap required, opened a new log
I20260619 10:29:58.243019 31255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a41961308f4460a416d37f0e6fdec5" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45883 } }
I20260619 10:29:58.243147 31255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:58.243178 31255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a41961308f4460a416d37f0e6fdec5, State: Initialized, Role: FOLLOWER
I20260619 10:29:58.243311 31255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a41961308f4460a416d37f0e6fdec5" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45883 } }
I20260619 10:29:58.243383 31255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:58.243418 31255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:58.243462 31255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:58.243988 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 31187
I20260619 10:29:58.244122 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:58.244326 9577 external_mini_cluster.cc:1428] Setting key 32be3b707e3de7e9f61786cf3f8e49d8
I20260619 10:29:58.244364 31255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a41961308f4460a416d37f0e6fdec5" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45883 } }
I20260619 10:29:58.244459 31255 leader_election.cc:302] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73a41961308f4460a416d37f0e6fdec5; no voters:
I20260619 10:29:58.244603 31255 leader_election.cc:288] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:58.244896 31255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:58.244622 31258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:58.245105 31258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [term 1 LEADER]: Becoming Leader. State: Replica: 73a41961308f4460a416d37f0e6fdec5, State: Running, Role: LEADER
I20260619 10:29:58.245222 31258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a41961308f4460a416d37f0e6fdec5" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45883 } }
I20260619 10:29:58.246393 31262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73a41961308f4460a416d37f0e6fdec5. Latest consensus state: current_term: 1 leader_uuid: "73a41961308f4460a416d37f0e6fdec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a41961308f4460a416d37f0e6fdec5" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45883 } } }
I20260619 10:29:58.246513 31262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:58.246515 31259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73a41961308f4460a416d37f0e6fdec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a41961308f4460a416d37f0e6fdec5" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45883 } } }
I20260619 10:29:58.246598 31259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5 [sys.catalog]: This master's current role is: LEADER
W20260619 10:29:58.246706 31275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:29:58.246749 31275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:29:58.246805 31277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:58.246958 31277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:58.248399 31277 catalog_manager.cc:1347] Generated new cluster ID: eeb9bef3c5ec4c6fa7043a431c51460b
I20260619 10:29:58.248449 31277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:58.249892 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:58.244757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39397 (local address 127.9.90.126:45883)
0619 10:29:58.244925 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:58.244934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:58.244966 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:29:58.245074 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:58.245083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:58.245149 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:58.245283 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:58.245552 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:58.247066 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:58.247988 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:58.248525 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:58.248617 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:58.249254 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:58.249258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:58.249258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:58.249425 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:58.249453 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:58.249463 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:58.249533 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:58.249620 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 10:29:58.249648 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260619 10:29:58.250705 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:58.254786 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:58.251912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60167 (local address 127.9.90.126:45883)
0619 10:29:58.251943 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:58.251944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:58.252037 (+ 93us) server_negotiation.cc:407] Connection header received
0619 10:29:58.252097 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:58.252098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:58.252107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:58.252150 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:58.252262 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:58.253024 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:58.253686 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:29:58.253941 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:29:58.253991 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:29:58.254567 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:58.254568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:58.254569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:58.254573 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:58.254587 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:58.254591 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:58.254639 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:58.254705 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 10:29:58.254716 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:29:58.264377 31277 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:58.264869 31277 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:58.275076 31277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73a41961308f4460a416d37f0e6fdec5: Generated new TSK 0
I20260619 10:29:58.275254 31277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:58.279532 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 31187
W20260619 10:29:58.282971 31172 connection.cc:535] client connection to 127.9.90.126:45883 recv error: Network error: failed to read from TLS socket (remote: 127.9.90.126:45883): Connection reset by peer (error 104)
2026-06-19T10:29:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:58 dist-test-slave-hw7q krb5kdc[31286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:58 dist-test-slave-hw7q krb5kdc[31286](info): set up 2 sockets
Jun 19 10:29:58 dist-test-slave-hw7q krb5kdc[31286](info): commencing operation
krb5kdc: starting...
W20260619 10:29:59.229609 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:29:59 dist-test-slave-hw7q krb5kdc[31286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:29:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:29:59.335815 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:41239
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:46537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:29:59.413123 31299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:29:59.413318 31299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:29:59.413339 31299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:29:59.414480 31299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:29:59.414521 31299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:29:59.414535 31299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:29:59.414548 31299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:29:59.414572 31299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:29:59.415964 31299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:46537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:41239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:41239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:29:59.416180 31299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:29:59.416373 31299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:29:59.420779 31307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:59.420779 31305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:29:59.420820 31304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:29:59.420940 31299 server_base.cc:1034] running on GCE node
I20260619 10:29:59.421273 31299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:29:59.421525 31299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:29:59.422662 31299 hybrid_clock.cc:648] HybridClock initialized: now 1781864999422648 us; error 30 us; skew 500 ppm
Jun 19 10:29:59 dist-test-slave-hw7q krb5kdc[31286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781864999, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:29:59.424860 31299 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:29:59.425283 31299 webserver.cc:460] Webserver started at http://127.9.90.126:33955/ using document root <none> and password file <none>
I20260619 10:29:59.425469 31299 fs_manager.cc:359] Metadata directory not provided
I20260619 10:29:59.425514 31299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:29:59.425632 31299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:29:59.426638 31299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "b54a2590f6e94debb6a70012a6000dba"
format_stamp: "Formatted at 2026-06-19 10:29:59 on dist-test-slave-hw7q"
server_key: "bc240ca54153816fcb38fce6f99059a5"
server_key_iv: "dea4ef570a29c083e73d189ec872bc6b"
server_key_version: "encryptionkey@0"
I20260619 10:29:59.426952 31299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "b54a2590f6e94debb6a70012a6000dba"
format_stamp: "Formatted at 2026-06-19 10:29:59 on dist-test-slave-hw7q"
server_key: "bc240ca54153816fcb38fce6f99059a5"
server_key_iv: "dea4ef570a29c083e73d189ec872bc6b"
server_key_version: "encryptionkey@0"
I20260619 10:29:59.428090 31299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:29:59.428691 31314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:29:59.428834 31299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:29:59.428910 31299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "b54a2590f6e94debb6a70012a6000dba"
format_stamp: "Formatted at 2026-06-19 10:29:59 on dist-test-slave-hw7q"
server_key: "bc240ca54153816fcb38fce6f99059a5"
server_key_iv: "dea4ef570a29c083e73d189ec872bc6b"
server_key_version: "encryptionkey@0"
I20260619 10:29:59.428974 31299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:29:59.432050 31299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:29:59.433636 31299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:29:59.433756 31299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:29:59.437218 31299 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:41239
I20260619 10:29:59.437245 31366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:41239 every 8 connection(s)
I20260619 10:29:59.437520 31299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:29:59.438421 31367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:29:59.439465 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 31299
I20260619 10:29:59.439564 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:29:59.439743 9577 external_mini_cluster.cc:1428] Setting key 960e268f6b79ab45e112d6ccd3ba738f
I20260619 10:29:59.440809 31367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba: Bootstrap starting.
I20260619 10:29:59.441452 31367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba: Neither blocks nor log segments found. Creating new log.
I20260619 10:29:59.441738 31367 log.cc:824] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba: Log is configured to *not* fsync() on all Append() calls
I20260619 10:29:59.442222 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:59.440189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46267 (local address 127.9.90.126:41239)
0619 10:29:59.440361 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:59.440371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:59.440404 (+ 33us) server_negotiation.cc:407] Connection header received
0619 10:29:59.440906 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:59.440915 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:59.440946 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:59.441089 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:59.441780 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:59.441804 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:59.441804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:59.441809 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:59.441852 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:59.441866 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:59.441946 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:59.442002 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 10:29:59.442041 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260619 10:29:59.442458 31367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba: No bootstrap required, opened a new log
I20260619 10:29:59.443121 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:29:59.444432 31367 raft_consensus.cc:348] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54a2590f6e94debb6a70012a6000dba" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41239 } }
I20260619 10:29:59.444571 31367 raft_consensus.cc:374] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:29:59.444602 31367 raft_consensus.cc:729] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54a2590f6e94debb6a70012a6000dba, State: Initialized, Role: FOLLOWER
I20260619 10:29:59.444711 31367 consensus_queue.cc:260] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54a2590f6e94debb6a70012a6000dba" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41239 } }
I20260619 10:29:59.444778 31367 raft_consensus.cc:388] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:29:59.444833 31367 raft_consensus.cc:482] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:29:59.444882 31367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:29:59.445355 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:29:59.444375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35853 (local address 127.9.90.126:41239)
0619 10:29:59.444407 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:29:59.444409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:29:59.444512 (+ 103us) server_negotiation.cc:407] Connection header received
0619 10:29:59.444573 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:29:59.444574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:29:59.444578 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:29:59.444624 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:29:59.445177 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:29:59.445178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:29:59.445178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:29:59.445182 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:29:59.445195 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:29:59.445198 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:29:59.445253 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 10:29:59.445287 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 10:29:59.445299 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:29:59.445406 31367 raft_consensus.cc:504] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54a2590f6e94debb6a70012a6000dba" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41239 } }
I20260619 10:29:59.445502 31367 leader_election.cc:302] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b54a2590f6e94debb6a70012a6000dba; no voters:
I20260619 10:29:59.445643 31367 leader_election.cc:288] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:29:59.445669 31377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:29:59.445813 31377 raft_consensus.cc:686] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [term 1 LEADER]: Becoming Leader. State: Replica: b54a2590f6e94debb6a70012a6000dba, State: Running, Role: LEADER
I20260619 10:29:59.445884 31367 sys_catalog.cc:564] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:29:59.445894 31377 consensus_queue.cc:237] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54a2590f6e94debb6a70012a6000dba" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41239 } }
I20260619 10:29:59.446259 31379 sys_catalog.cc:455] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b54a2590f6e94debb6a70012a6000dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54a2590f6e94debb6a70012a6000dba" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41239 } } }
I20260619 10:29:59.446381 31379 sys_catalog.cc:458] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:59.446556 31378 sys_catalog.cc:455] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader b54a2590f6e94debb6a70012a6000dba. Latest consensus state: current_term: 1 leader_uuid: "b54a2590f6e94debb6a70012a6000dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54a2590f6e94debb6a70012a6000dba" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 41239 } } }
I20260619 10:29:59.446657 31378 sys_catalog.cc:458] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba [sys.catalog]: This master's current role is: LEADER
I20260619 10:29:59.446810 31386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:29:59.447232 31386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:29:59.448593 31386 catalog_manager.cc:1347] Generated new cluster ID: 6c34811bca7946ed95c760286060349b
I20260619 10:29:59.448642 31386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:29:59.471380 31386 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:29:59.471570 31386 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:29:59.475839 31386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b54a2590f6e94debb6a70012a6000dba: Generated new TSK 0
I20260619 10:29:59.476009 31386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:29:59.479723 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 31299
2026-06-19T10:29:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:29:59 dist-test-slave-hw7q krb5kdc[31398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:29:59 dist-test-slave-hw7q krb5kdc[31398](info): set up 2 sockets
Jun 19 10:29:59 dist-test-slave-hw7q krb5kdc[31398](info): commencing operation
krb5kdc: starting...
W20260619 10:30:00.437522 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:30:00 dist-test-slave-hw7q krb5kdc[31398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:30:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:30:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:30:00.546147 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:37909
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:37909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:30:00.624380 31411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:30:00.624567 31411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:30:00.624588 31411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:30:00.625633 31411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:30:00.625665 31411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:30:00.625677 31411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:30:00.625689 31411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:30:00.625703 31411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:30:00.627061 31411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:37909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:37909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:30:00.627249 31411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:30:00.627406 31411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:30:00.631783 31419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:00.631903 31411 server_base.cc:1034] running on GCE node
W20260619 10:30:00.631790 31416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:00.631799 31417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:00.632221 31411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:30:00.632488 31411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:30:00.633631 31411 hybrid_clock.cc:648] HybridClock initialized: now 1781865000633613 us; error 28 us; skew 500 ppm
Jun 19 10:30:00 dist-test-slave-hw7q krb5kdc[31398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865000, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:30:00.635892 31411 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:30:00.636320 31411 webserver.cc:460] Webserver started at http://127.9.90.126:35083/ using document root <none> and password file <none>
I20260619 10:30:00.636507 31411 fs_manager.cc:359] Metadata directory not provided
I20260619 10:30:00.636554 31411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:30:00.636673 31411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:30:00.637650 31411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "a1772887d32144b79577e7a67ddc1859"
format_stamp: "Formatted at 2026-06-19 10:30:00 on dist-test-slave-hw7q"
server_key: "c61a8098b768975c370e6d5818f860d6"
server_key_iv: "7918e6d2bc08c9b7dfe8204c739d7905"
server_key_version: "encryptionkey@0"
I20260619 10:30:00.637984 31411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "a1772887d32144b79577e7a67ddc1859"
format_stamp: "Formatted at 2026-06-19 10:30:00 on dist-test-slave-hw7q"
server_key: "c61a8098b768975c370e6d5818f860d6"
server_key_iv: "7918e6d2bc08c9b7dfe8204c739d7905"
server_key_version: "encryptionkey@0"
I20260619 10:30:00.639160 31411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:30:00.639752 31426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:30:00.639894 31411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:30:00.639971 31411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "a1772887d32144b79577e7a67ddc1859"
format_stamp: "Formatted at 2026-06-19 10:30:00 on dist-test-slave-hw7q"
server_key: "c61a8098b768975c370e6d5818f860d6"
server_key_iv: "7918e6d2bc08c9b7dfe8204c739d7905"
server_key_version: "encryptionkey@0"
I20260619 10:30:00.640028 31411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:30:00.653489 31411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:30:00.655208 31411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:30:00.655339 31411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:30:00.658917 31411 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:37909
I20260619 10:30:00.658951 31478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:37909 every 8 connection(s)
I20260619 10:30:00.659279 31411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:30:00.660048 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:30:00.660290 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 31411
I20260619 10:30:00.660386 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:30:00.660539 9577 external_mini_cluster.cc:1428] Setting key ec30aab29d42bd761d24477232d24afc
I20260619 10:30:00.662773 31479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859: Bootstrap starting.
I20260619 10:30:00.663324 31479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859: Neither blocks nor log segments found. Creating new log.
I20260619 10:30:00.663581 31479 log.cc:824] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859: Log is configured to *not* fsync() on all Append() calls
I20260619 10:30:00.664273 31479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859: No bootstrap required, opened a new log
I20260619 10:30:00.666541 31479 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1772887d32144b79577e7a67ddc1859" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37909 } }
I20260619 10:30:00.666683 31479 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:30:00.666705 31479 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1772887d32144b79577e7a67ddc1859, State: Initialized, Role: FOLLOWER
I20260619 10:30:00.666810 31479 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1772887d32144b79577e7a67ddc1859" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37909 } }
I20260619 10:30:00.666873 31479 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:30:00.666893 31479 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:30:00.666939 31479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 10:30:00 dist-test-slave-hw7q krb5kdc[31398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:30:00.667476 31479 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1772887d32144b79577e7a67ddc1859" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37909 } }
I20260619 10:30:00.667577 31479 leader_election.cc:302] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1772887d32144b79577e7a67ddc1859; no voters:
I20260619 10:30:00.667738 31479 leader_election.cc:288] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:30:00.667778 31484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:30:00.667910 31484 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [term 1 LEADER]: Becoming Leader. State: Replica: a1772887d32144b79577e7a67ddc1859, State: Running, Role: LEADER
I20260619 10:30:00.667984 31479 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:30:00.667997 31484 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1772887d32144b79577e7a67ddc1859" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37909 } }
I20260619 10:30:00.668324 31484 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1772887d32144b79577e7a67ddc1859. Latest consensus state: current_term: 1 leader_uuid: "a1772887d32144b79577e7a67ddc1859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1772887d32144b79577e7a67ddc1859" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37909 } } }
I20260619 10:30:00.668390 31484 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:00.668650 31485 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1772887d32144b79577e7a67ddc1859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1772887d32144b79577e7a67ddc1859" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 37909 } } }
I20260619 10:30:00.668762 31485 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859 [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:00.669077 31488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:30:00.669567 31488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:30:00.671047 31488 catalog_manager.cc:1347] Generated new cluster ID: d62011bb88ed4cc693c0a7332c2fd38b
I20260619 10:30:00.671098 31488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:30:00.671672 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:00.661006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44329 (local address 127.9.90.126:37909)
0619 10:30:00.661162 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:00.661172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:00.661197 (+ 25us) server_negotiation.cc:407] Connection header received
0619 10:30:00.661600 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:00.661606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:00.661677 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:00.661821 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:00.662561 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:00.663479 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:00.664419 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:00.664814 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:00.664893 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:00.667524 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:00.667528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:00.667529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:00.667541 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:00.670025 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:00.670395 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:00.670398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:00.670399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:00.670495 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:00.670697 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:00.670698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:00.670699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:00.670986 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:00.671100 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:00.671363 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 10:30:00.671487 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260619 10:30:00.672763 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:30:00.677739 31488 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:30:00.678393 31488 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:30:00.680850 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:00.674072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60999 (local address 127.9.90.126:37909)
0619 10:30:00.674101 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:00.674103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:00.674319 (+ 216us) server_negotiation.cc:407] Connection header received
0619 10:30:00.674391 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:00.674391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:00.674400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:00.674425 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:00.674989 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:00.675706 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:00.676656 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:00.676959 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:00.676994 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:00.678409 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:00.678411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:00.678411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:00.678418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:00.679803 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:00.680123 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:00.680124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:00.680124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:00.680178 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:00.680335 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:00.680336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:00.680336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:00.680433 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:00.680491 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:00.680674 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 10:30:00.680766 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 10:30:00.687517 31488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1772887d32144b79577e7a67ddc1859: Generated new TSK 0
I20260619 10:30:00.687711 31488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:30:00.694499 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 31411
2026-06-19T10:30:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:30:00 dist-test-slave-hw7q krb5kdc[31510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:30:00 dist-test-slave-hw7q krb5kdc[31510](info): set up 2 sockets
Jun 19 10:30:00 dist-test-slave-hw7q krb5kdc[31510](info): commencing operation
krb5kdc: starting...
W20260619 10:30:01.645605 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:30:01 dist-test-slave-hw7q krb5kdc[31510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:30:01.754017 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45935
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:30:01.833595 31523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:30:01.833863 31523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:30:01.833894 31523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:30:01.835005 31523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:30:01.835042 31523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:30:01.835054 31523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:30:01.835075 31523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:30:01.835098 31523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:30:01.836501 31523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:45935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:45935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:30:01.836714 31523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:30:01.836901 31523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:30:01.841413 31529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:01.841385 31531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:01.841385 31528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:01.841380 31523 server_base.cc:1034] running on GCE node
I20260619 10:30:01.841887 31523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:30:01.842172 31523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:30:01.843338 31523 hybrid_clock.cc:648] HybridClock initialized: now 1781865001843317 us; error 30 us; skew 500 ppm
Jun 19 10:30:01 dist-test-slave-hw7q krb5kdc[31510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865001, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:30:01.845773 31523 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:30:01.846227 31523 webserver.cc:460] Webserver started at http://127.9.90.126:42703/ using document root <none> and password file <none>
I20260619 10:30:01.846414 31523 fs_manager.cc:359] Metadata directory not provided
I20260619 10:30:01.846462 31523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:30:01.846581 31523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:30:01.847560 31523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "08d83f56e82f4c64a403799e3a566db7"
format_stamp: "Formatted at 2026-06-19 10:30:01 on dist-test-slave-hw7q"
server_key: "b49c476ad043df2d8bc9bc18fb3db6ff"
server_key_iv: "3c7f8548efad08f0cac12e330b74f2dc"
server_key_version: "encryptionkey@0"
I20260619 10:30:01.847908 31523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "08d83f56e82f4c64a403799e3a566db7"
format_stamp: "Formatted at 2026-06-19 10:30:01 on dist-test-slave-hw7q"
server_key: "b49c476ad043df2d8bc9bc18fb3db6ff"
server_key_iv: "3c7f8548efad08f0cac12e330b74f2dc"
server_key_version: "encryptionkey@0"
I20260619 10:30:01.849092 31523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 10:30:01.849825 31538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:30:01.849994 31523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 10:30:01.850076 31523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "08d83f56e82f4c64a403799e3a566db7"
format_stamp: "Formatted at 2026-06-19 10:30:01 on dist-test-slave-hw7q"
server_key: "b49c476ad043df2d8bc9bc18fb3db6ff"
server_key_iv: "3c7f8548efad08f0cac12e330b74f2dc"
server_key_version: "encryptionkey@0"
I20260619 10:30:01.850139 31523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:30:01.864289 31523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:30:01.865988 31523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:30:01.866117 31523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:30:01.869622 31523 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:45935
I20260619 10:30:01.869663 31590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:45935 every 8 connection(s)
I20260619 10:30:01.869998 31523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:30:01.870822 31591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:30:01.873000 31591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7: Bootstrap starting.
I20260619 10:30:01.873538 31591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7: Neither blocks nor log segments found. Creating new log.
I20260619 10:30:01.873800 31591 log.cc:824] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7: Log is configured to *not* fsync() on all Append() calls
I20260619 10:30:01.874439 31591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7: No bootstrap required, opened a new log
I20260619 10:30:01.875947 31591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d83f56e82f4c64a403799e3a566db7" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45935 } }
I20260619 10:30:01.876081 31591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:30:01.876101 31591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d83f56e82f4c64a403799e3a566db7, State: Initialized, Role: FOLLOWER
I20260619 10:30:01.876253 31591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d83f56e82f4c64a403799e3a566db7" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45935 } }
I20260619 10:30:01.876328 31591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:30:01.876351 31591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:30:01.876380 31591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:30:01.877135 31591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d83f56e82f4c64a403799e3a566db7" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45935 } }
I20260619 10:30:01.877224 31591 leader_election.cc:302] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08d83f56e82f4c64a403799e3a566db7; no voters:
I20260619 10:30:01.877367 31591 leader_election.cc:288] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:30:01.877432 31594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:30:01.877578 31591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:30:01.877624 31594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [term 1 LEADER]: Becoming Leader. State: Replica: 08d83f56e82f4c64a403799e3a566db7, State: Running, Role: LEADER
I20260619 10:30:01.877753 31594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d83f56e82f4c64a403799e3a566db7" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45935 } }
I20260619 10:30:01.878175 31596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08d83f56e82f4c64a403799e3a566db7. Latest consensus state: current_term: 1 leader_uuid: "08d83f56e82f4c64a403799e3a566db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d83f56e82f4c64a403799e3a566db7" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45935 } } }
I20260619 10:30:01.878278 31595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08d83f56e82f4c64a403799e3a566db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d83f56e82f4c64a403799e3a566db7" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 45935 } } }
I20260619 10:30:01.878371 31595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:01.878474 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 31523
I20260619 10:30:01.878578 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:30:01.878278 31596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7 [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:01.878767 31603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:30:01.878808 9577 external_mini_cluster.cc:1428] Setting key 9eb66d40fa69f507a1e39632d1179cd5
I20260619 10:30:01.879400 31603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:30:01.880854 31603 catalog_manager.cc:1347] Generated new cluster ID: a782423cc77f4df2a69dfe34e5298064
I20260619 10:30:01.880911 31603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:30:01.885164 31613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:01.879239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49589 (local address 127.9.90.126:45935)
0619 10:30:01.879404 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:01.879413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:01.879616 (+ 203us) server_negotiation.cc:407] Connection header received
0619 10:30:01.879937 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:01.879946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:01.880004 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:01.880150 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:01.880374 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:01.881783 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:01.882888 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:01.883526 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:01.883627 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:01.884315 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:01.884319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:01.884319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:30:01.884536 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:01.884580 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:30:01.884594 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:01.884659 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:01.884735 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 10:30:01.884793 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260619 10:30:01.886041 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:30:01.891263 31613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:01.887470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40423 (local address 127.9.90.126:45935)
0619 10:30:01.887505 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:01.887506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:01.887776 (+ 270us) server_negotiation.cc:407] Connection header received
0619 10:30:01.887868 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:01.887869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:01.887880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:01.887908 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:01.888095 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:01.888809 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:01.889841 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:01.890195 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:01.890236 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:01.891012 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:01.891013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:01.891013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:30:01.891018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:01.891032 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:30:01.891036 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:01.891063 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:01.891162 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 10:30:01.891186 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:30:01.901355 31603 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:30:01.901990 31603 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:30:01.910709 31603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08d83f56e82f4c64a403799e3a566db7: Generated new TSK 0
I20260619 10:30:01.910879 31603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:30:01.916780 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 31523
2026-06-19T10:30:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:30:01 dist-test-slave-hw7q krb5kdc[31622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:30:01 dist-test-slave-hw7q krb5kdc[31622](info): set up 2 sockets
Jun 19 10:30:01 dist-test-slave-hw7q krb5kdc[31622](info): commencing operation
krb5kdc: starting...
W20260619 10:30:02.861720 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:30:02 dist-test-slave-hw7q krb5kdc[31622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:30:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:30:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:30:02.972450 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:35941
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:35941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:30:03.051739 31635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:30:03.051944 31635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:30:03.051982 31635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:30:03.053090 31635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:30:03.053128 31635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:30:03.053148 31635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:30:03.053162 31635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:30:03.053179 31635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:30:03.054646 31635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:35941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:35941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:30:03.054870 31635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:30:03.055073 31635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:30:03.059494 31643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:03.059995 31641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:03.060004 31640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:03.060189 31635 server_base.cc:1034] running on GCE node
I20260619 10:30:03.060400 31635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:30:03.060668 31635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:30:03.061872 31635 hybrid_clock.cc:648] HybridClock initialized: now 1781865003061850 us; error 35 us; skew 500 ppm
Jun 19 10:30:03 dist-test-slave-hw7q krb5kdc[31622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865003, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:30:03.067384 31635 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:30:03.067811 31635 webserver.cc:460] Webserver started at http://127.9.90.126:45319/ using document root <none> and password file <none>
I20260619 10:30:03.068035 31635 fs_manager.cc:359] Metadata directory not provided
I20260619 10:30:03.068087 31635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:30:03.068209 31635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:30:03.069233 31635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58"
format_stamp: "Formatted at 2026-06-19 10:30:03 on dist-test-slave-hw7q"
server_key: "68736a011497b8766e6e11f872e58a84"
server_key_iv: "99b547b3c553647236f82370a37d61c7"
server_key_version: "encryptionkey@0"
I20260619 10:30:03.069567 31635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58"
format_stamp: "Formatted at 2026-06-19 10:30:03 on dist-test-slave-hw7q"
server_key: "68736a011497b8766e6e11f872e58a84"
server_key_iv: "99b547b3c553647236f82370a37d61c7"
server_key_version: "encryptionkey@0"
I20260619 10:30:03.070840 31635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:30:03.071579 31650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:30:03.071780 31635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:30:03.071864 31635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58"
format_stamp: "Formatted at 2026-06-19 10:30:03 on dist-test-slave-hw7q"
server_key: "68736a011497b8766e6e11f872e58a84"
server_key_iv: "99b547b3c553647236f82370a37d61c7"
server_key_version: "encryptionkey@0"
I20260619 10:30:03.071933 31635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:30:03.075142 31635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:30:03.076848 31635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:30:03.076968 31635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:30:03.080636 31635 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:35941
I20260619 10:30:03.080685 31702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:35941 every 8 connection(s)
I20260619 10:30:03.080976 31635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:30:03.081867 31703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:30:03.083950 31703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58: Bootstrap starting.
I20260619 10:30:03.084582 31703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58: Neither blocks nor log segments found. Creating new log.
I20260619 10:30:03.084810 31703 log.cc:824] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58: Log is configured to *not* fsync() on all Append() calls
I20260619 10:30:03.085436 31703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58: No bootstrap required, opened a new log
I20260619 10:30:03.086549 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 31635
I20260619 10:30:03.086668 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:30:03.086851 9577 external_mini_cluster.cc:1428] Setting key 4259402b3ebd925c44443bd258cfa0ae
I20260619 10:30:03.087348 31703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35941 } }
I20260619 10:30:03.087520 31703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 10:30:03.087553 31703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa7697d4bd34e0ebee4ba25a4f91e58, State: Initialized, Role: FOLLOWER
I20260619 10:30:03.087675 31703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35941 } }
I20260619 10:30:03.087740 31703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 10:30:03.087771 31703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 10:30:03.087816 31703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:30:03.088824 31703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35941 } }
I20260619 10:30:03.088876 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:03.087363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35011 (local address 127.9.90.126:35941)
0619 10:30:03.087524 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:03.087533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:03.087567 (+ 34us) server_negotiation.cc:407] Connection header received
0619 10:30:03.087684 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:03.087691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:03.087722 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:03.087861 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:03.088445 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:03.088447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:03.088448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:30:03.088451 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:03.088496 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:30:03.088507 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:03.088556 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:03.088657 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 10:30:03.088693 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260619 10:30:03.088912 31703 leader_election.cc:302] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fa7697d4bd34e0ebee4ba25a4f91e58; no voters:
I20260619 10:30:03.089056 31703 leader_election.cc:288] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 10:30:03.089108 31708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:30:03.089243 31708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa7697d4bd34e0ebee4ba25a4f91e58, State: Running, Role: LEADER
I20260619 10:30:03.089370 31708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35941 } }
I20260619 10:30:03.089453 31703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:30:03.089710 31708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35941 } } }
I20260619 10:30:03.089756 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:30:03.089838 31708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:03.089829 31709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fa7697d4bd34e0ebee4ba25a4f91e58. Latest consensus state: current_term: 1 leader_uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa7697d4bd34e0ebee4ba25a4f91e58" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 35941 } } }
I20260619 10:30:03.089962 31709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58 [sys.catalog]: This master's current role is: LEADER
W20260619 10:30:03.091140 31724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:30:03.091275 31724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:30:03.091346 31713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:30:03.091511 31713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:30:03.092759 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:03.091100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37035 (local address 127.9.90.126:35941)
0619 10:30:03.091130 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:03.091132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:03.091660 (+ 528us) server_negotiation.cc:407] Connection header received
0619 10:30:03.091807 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:03.091808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:03.091813 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:03.091858 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:03.092533 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:03.092534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:03.092534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 10:30:03.092538 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:03.092551 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 10:30:03.092555 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:03.092616 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:03.092690 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 10:30:03.092703 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:30:03.093191 31713 catalog_manager.cc:1347] Generated new cluster ID: ed8d40dc63e24544baa0d3ada0bc3247
I20260619 10:30:03.093245 31713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:30:03.104471 31713 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:30:03.104645 31713 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:30:03.117895 31713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fa7697d4bd34e0ebee4ba25a4f91e58: Generated new TSK 0
I20260619 10:30:03.118116 31713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 10:30:03.124251 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 31635
2026-06-19T10:30:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1207 ms)
[----------] 18 tests from AuthTokenIssuingTest (21610 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:30:03 dist-test-slave-hw7q krb5kdc[31734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:30:03 dist-test-slave-hw7q krb5kdc[31734](info): set up 2 sockets
Jun 19 10:30:03 dist-test-slave-hw7q krb5kdc[31734](info): commencing operation
krb5kdc: starting...
W20260619 10:30:04.069942 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:30:04.178617 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40493
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:40493,127.9.90.125:46437,127.9.90.124:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:30:04.255321 31747 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:30:04.255520 31747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:30:04.255542 31747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:30:04.255568 31747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:30:04.256683 31747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:30:04.256726 31747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:30:04.256743 31747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:30:04.256759 31747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:30:04.256774 31747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:30:04.258239 31747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:40493,127.9.90.125:46437,127.9.90.124:33253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:30:04.258471 31747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:30:04.258684 31747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:30:04.263113 31755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:04.263113 31753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:04.263217 31752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:04.263368 31747 server_base.cc:1034] running on GCE node
I20260619 10:30:04.263557 31747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:30:04.263815 31747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:30:04.264947 31747 hybrid_clock.cc:648] HybridClock initialized: now 1781865004264934 us; error 28 us; skew 500 ppm
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:30:04.267191 31747 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:30:04.267583 31747 webserver.cc:460] Webserver started at http://127.9.90.126:33237/ using document root <none> and password file <none>
I20260619 10:30:04.267772 31747 fs_manager.cc:359] Metadata directory not provided
I20260619 10:30:04.267810 31747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:30:04.267894 31747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:30:04.268813 31747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "a7bb84051e57447b9eaa8b69c5a31888"
format_stamp: "Formatted at 2026-06-19 10:30:04 on dist-test-slave-hw7q"
server_key: "b2117ab37c9ff3ced211fbe67e16c0ea"
server_key_iv: "586e0a5f38f763c18280740ebf48057b"
server_key_version: "encryptionkey@0"
I20260619 10:30:04.269078 31747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "a7bb84051e57447b9eaa8b69c5a31888"
format_stamp: "Formatted at 2026-06-19 10:30:04 on dist-test-slave-hw7q"
server_key: "b2117ab37c9ff3ced211fbe67e16c0ea"
server_key_iv: "586e0a5f38f763c18280740ebf48057b"
server_key_version: "encryptionkey@0"
I20260619 10:30:04.270234 31747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 10:30:04.270818 31762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:30:04.270977 31747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 10:30:04.271056 31747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "a7bb84051e57447b9eaa8b69c5a31888"
format_stamp: "Formatted at 2026-06-19 10:30:04 on dist-test-slave-hw7q"
server_key: "b2117ab37c9ff3ced211fbe67e16c0ea"
server_key_iv: "586e0a5f38f763c18280740ebf48057b"
server_key_version: "encryptionkey@0"
I20260619 10:30:04.271101 31747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:30:04.292177 31747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:30:04.294349 31747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:30:04.294456 31747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:30:04.298357 31747 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:40493
I20260619 10:30:04.298388 31814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:40493 every 8 connection(s)
I20260619 10:30:04.298748 31747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:30:04.299651 31815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:30:04.301151 31815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:04.302796 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 31747
I20260619 10:30:04.302897 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:30:04.303069 9577 external_mini_cluster.cc:1428] Setting key 983b509956b5d9e4f83bd1cc543ceac0
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.90.125@KRBTEST.COM; defaulting to no policy
I20260619 10:30:04.310829 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.301817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.0.0.1:41252)
0619 10:30:04.301969 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.302009 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:04.302448 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:04.302728 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:04.302731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:04.303043 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:04.303276 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.303282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.304276 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.304277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.304936 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.304938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.305557 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.305559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.305618 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.306082 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:04.306090 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:04.307594 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:04.309370 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.309376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.309378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.309554 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.309828 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.309829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.309830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.309909 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.310325 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:04.310328 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:04.310379 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 10:30:04.310466 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 10:30:04.310593 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260619 10:30:04.310837 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.301858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.9.90.126:40493)
0619 10:30:04.302008 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.302018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.302110 (+ 92us) server_negotiation.cc:407] Connection header received
0619 10:30:04.302594 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.302596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.302613 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.302695 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.303376 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.304176 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.305027 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.305457 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.305556 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.307687 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.307695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.307695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.307702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.309285 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.309644 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.309647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.309648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.309740 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.309996 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.309997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.309997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.310263 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.310339 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.310517 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.310672 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260619 10:30:04.311533 31815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:04.311806 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.311704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46437 (local address 127.0.0.1:56940)
0619 10:30:04.311749 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.311789 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 10:30:04.311930 31766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
I20260619 10:30:04.312309 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.312235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46437 (local address 127.0.0.1:56952)
0619 10:30:04.312270 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.312293 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 10:30:04.312410 31815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.125:46437: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab.
Entry for principal kudu/127.9.90.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab.
I20260619 10:30:04.326337 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.125:46437
--webserver_interface=127.9.90.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab
--principal=kudu/127.9.90.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:40493,127.9.90.125:46437,127.9.90.124:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:30:04.361690 31815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } attempt: 1
I20260619 10:30:04.362195 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.362066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46437 (local address 127.0.0.1:56958)
0619 10:30:04.362127 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.362161 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 10:30:04.362565 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.362484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46437 (local address 127.0.0.1:56964)
0619 10:30:04.362525 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.362550 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 10:30:04.362664 31815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.125:46437: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
W20260619 10:30:04.403573 31822 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:30:04.403779 31822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:30:04.403801 31822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:30:04.403824 31822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:30:04.404938 31822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:30:04.404978 31822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:30:04.404995 31822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:30:04.405012 31822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:30:04.405026 31822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:30:04.406494 31822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:40493,127.9.90.125:46437,127.9.90.124:33253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab
--principal=kudu/127.9.90.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.125:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.90.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:30:04.406730 31822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:30:04.406939 31822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:30:04.411461 31827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:04.411469 31830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:04.411614 31822 server_base.cc:1034] running on GCE node
W20260619 10:30:04.411489 31828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:04.411952 31822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:30:04.412238 31822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:30:04.413376 31822 hybrid_clock.cc:648] HybridClock initialized: now 1781865004413363 us; error 27 us; skew 500 ppm
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:30:04.415810 31822 init.cc:377] Logged in from keytab as kudu/127.9.90.125@KRBTEST.COM (short username kudu)
I20260619 10:30:04.416255 31822 webserver.cc:460] Webserver started at http://127.9.90.125:38447/ using document root <none> and password file <none>
I20260619 10:30:04.416473 31822 fs_manager.cc:359] Metadata directory not provided
I20260619 10:30:04.416519 31822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:30:04.416653 31822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:30:04.417687 31822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/data/instance:
uuid: "af907bfb06954d10b9710b6ae0cf1b69"
format_stamp: "Formatted at 2026-06-19 10:30:04 on dist-test-slave-hw7q"
server_key: "38485824384328c50395ee56cd3368f2"
server_key_iv: "f1b41777225833b6ab28c29878ebf217"
server_key_version: "encryptionkey@0"
I20260619 10:30:04.418200 31822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/wal/instance:
uuid: "af907bfb06954d10b9710b6ae0cf1b69"
format_stamp: "Formatted at 2026-06-19 10:30:04 on dist-test-slave-hw7q"
server_key: "38485824384328c50395ee56cd3368f2"
server_key_iv: "f1b41777225833b6ab28c29878ebf217"
server_key_version: "encryptionkey@0"
I20260619 10:30:04.419457 31822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:30:04.420254 31837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:30:04.420413 31822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:30:04.420504 31822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/wal
uuid: "af907bfb06954d10b9710b6ae0cf1b69"
format_stamp: "Formatted at 2026-06-19 10:30:04 on dist-test-slave-hw7q"
server_key: "38485824384328c50395ee56cd3368f2"
server_key_iv: "f1b41777225833b6ab28c29878ebf217"
server_key_version: "encryptionkey@0"
I20260619 10:30:04.420620 31822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:30:04.430804 31815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } attempt: 2
I20260619 10:30:04.431169 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.431041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46437 (local address 127.0.0.1:56968)
0619 10:30:04.431097 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.431135 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:30:04.431505 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.431415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46437 (local address 127.0.0.1:56982)
0619 10:30:04.431458 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.431484 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:30:04.431592 31815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.125:46437: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46437: connect: Connection refused (error 111)
I20260619 10:30:04.441406 31822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:30:04.443369 31822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:30:04.443471 31822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:30:04.447012 31822 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.125:46437
I20260619 10:30:04.447055 31889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.125:46437 every 8 connection(s)
I20260619 10:30:04.447361 31822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
I20260619 10:30:04.448251 31890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:30:04.449671 31890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:04.450500 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 31822
I20260619 10:30:04.450590 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-1/wal/instance
I20260619 10:30:04.450755 9577 external_mini_cluster.cc:1428] Setting key 1262720e126902ef29bfc47ce71942d8
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.90.124@KRBTEST.COM; defaulting to no policy
I20260619 10:30:04.458582 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.450324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.9.90.126:40493)
0619 10:30:04.450377 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.450379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.450584 (+ 205us) server_negotiation.cc:407] Connection header received
0619 10:30:04.451110 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.451112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.451126 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.451183 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.451753 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.452225 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.453449 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.453661 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.453712 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.455982 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.455983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.455984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.455990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.457214 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.457673 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.457675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.457675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.457737 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.457989 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.457990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.457990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.458096 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.458152 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.458414 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.458491 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 10:30:04.458766 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.450293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.0.0.1:41254)
0619 10:30:04.450450 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.450485 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:04.450983 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:04.451251 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:04.451255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:04.451476 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:04.451659 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.451663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.452310 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.452310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.453352 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.453355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.453758 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.453759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.453926 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.454369 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:04.454378 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:04.455875 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:04.457301 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.457308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.457311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.457582 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.457821 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.457822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.457822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.457922 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.458165 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:04.458171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:04.458292 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 10:30:04.458380 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 10:30:04.458561 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260619 10:30:04.459317 31890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.124@KRBTEST.COM" created.
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
I20260619 10:30:04.466432 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.459565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46437 (local address 127.0.0.1:56984)
0619 10:30:04.459609 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.459618 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:04.459651 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:04.459956 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:04.459957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:04.460078 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:04.460134 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.460136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.460990 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.460991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.461610 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.461612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.462233 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.462233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.462283 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.462626 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:04.462631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:04.463893 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:04.465380 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.465381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.465382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.465572 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.465755 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.465756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.465756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.465835 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.466082 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:04.466082 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:04.466125 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 10:30:04.466198 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 10:30:04.466293 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:30:04.466459 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.459643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.9.90.125:46437)
0619 10:30:04.459805 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.459809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.459832 (+ 23us) server_negotiation.cc:407] Connection header received
0619 10:30:04.459853 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.459855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.459868 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.459940 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.460209 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.460904 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.461705 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.462169 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.462236 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.463968 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.463975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.463975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.463980 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.465291 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.465644 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.465646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.465646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.465692 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.465893 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.465894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.465894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.466015 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.466075 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.466225 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.466338 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260619 10:30:04.467047 31890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:04.467310 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.467225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50490)
0619 10:30:04.467253 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.467293 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 10:30:04.467442 31839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
I20260619 10:30:04.467806 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.467732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50498)
0619 10:30:04.467755 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.467777 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 10:30:04.467898 31890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:33253: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab.
Entry for principal kudu/127.9.90.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab.
I20260619 10:30:04.476372 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.124:33253
--webserver_interface=127.9.90.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab
--principal=kudu/127.9.90.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:40493,127.9.90.125:46437,127.9.90.124:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:30:04.517104 31890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } attempt: 1
I20260619 10:30:04.517547 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.517430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50506)
0619 10:30:04.517486 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.517519 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 10:30:04.517978 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.517843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50516)
0619 10:30:04.517905 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.517947 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 10:30:04.518086 31890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:33253: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
I20260619 10:30:04.522725 31815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } attempt: 3
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
I20260619 10:30:04.529722 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.523023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46437 (local address 127.0.0.1:56986)
0619 10:30:04.523078 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.523089 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:04.523183 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:04.523353 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:04.523353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:04.523495 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:04.523557 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.523559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.524447 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.524448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.524968 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.524969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.525442 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.525442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.525493 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.525921 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:04.525925 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:04.527287 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:04.528882 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.528883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.528884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.529032 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.529220 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.529221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.529221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.529267 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.529502 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:04.529503 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:04.529538 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 10:30:04.529581 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 10:30:04.529649 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 10:30:04.529767 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.523078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.9.90.125:46437)
0619 10:30:04.523145 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.523147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.523159 (+ 12us) server_negotiation.cc:407] Connection header received
0619 10:30:04.523249 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.523250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.523262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.523343 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.523621 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.524368 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.525044 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.525375 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.525443 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.527374 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.527376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.527376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.527383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.528803 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.529099 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.529099 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.529100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.529162 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.529320 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.529321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.529321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.529448 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.529510 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.529605 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.529678 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 10:30:04.530192 31815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:04.530479 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.530390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50526)
0619 10:30:04.530434 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.530459 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 10:30:04.530789 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.530719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50528)
0619 10:30:04.530754 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.530775 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 10:30:04.530889 31815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:33253: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
W20260619 10:30:04.557175 31897 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:30:04.557366 31897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:30:04.557381 31897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:30:04.557399 31897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:30:04.558503 31897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:30:04.558540 31897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:30:04.558552 31897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:30:04.558563 31897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:30:04.558573 31897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:30:04.559949 31897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:40493,127.9.90.125:46437,127.9.90.124:33253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab
--principal=kudu/127.9.90.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.124:33253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.90.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:30:04.560134 31897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:30:04.560293 31897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:30:04.562011 31815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } attempt: 1
I20260619 10:30:04.562424 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.562313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50534)
0619 10:30:04.562370 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.562398 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 10:30:04.562759 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.562672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50536)
0619 10:30:04.562719 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.562741 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 10:30:04.562847 31815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:33253: Network error: Client connection negotiation failed: client connection to 127.9.90.124:33253: connect: Connection refused (error 111)
W20260619 10:30:04.564807 31903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:04.564818 31905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:04.565062 31902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:04.564935 31897 server_base.cc:1034] running on GCE node
I20260619 10:30:04.565295 31897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:30:04.565585 31897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:30:04.566725 31897 hybrid_clock.cc:648] HybridClock initialized: now 1781865004566710 us; error 29 us; skew 500 ppm
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:30:04.568951 31897 init.cc:377] Logged in from keytab as kudu/127.9.90.124@KRBTEST.COM (short username kudu)
I20260619 10:30:04.569351 31897 webserver.cc:460] Webserver started at http://127.9.90.124:42659/ using document root <none> and password file <none>
I20260619 10:30:04.569516 31897 fs_manager.cc:359] Metadata directory not provided
I20260619 10:30:04.569555 31897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:30:04.569643 31897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:30:04.570644 31897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/data/instance:
uuid: "cd54abf834984375b113d7c245ebe84f"
format_stamp: "Formatted at 2026-06-19 10:30:04 on dist-test-slave-hw7q"
server_key: "8df4a0889abccc1d372cb93a337e8026"
server_key_iv: "5537ff31a42648fe63b64bf39e095017"
server_key_version: "encryptionkey@0"
I20260619 10:30:04.570921 31897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/wal/instance:
uuid: "cd54abf834984375b113d7c245ebe84f"
format_stamp: "Formatted at 2026-06-19 10:30:04 on dist-test-slave-hw7q"
server_key: "8df4a0889abccc1d372cb93a337e8026"
server_key_iv: "5537ff31a42648fe63b64bf39e095017"
server_key_version: "encryptionkey@0"
I20260619 10:30:04.572063 31897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 10:30:04.572674 31912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:30:04.572811 31897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:30:04.572880 31897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/wal
uuid: "cd54abf834984375b113d7c245ebe84f"
format_stamp: "Formatted at 2026-06-19 10:30:04 on dist-test-slave-hw7q"
server_key: "8df4a0889abccc1d372cb93a337e8026"
server_key_iv: "5537ff31a42648fe63b64bf39e095017"
server_key_version: "encryptionkey@0"
I20260619 10:30:04.572952 31897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:30:04.580912 31897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:30:04.582844 31897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:30:04.582947 31897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:30:04.586249 31890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } attempt: 2
I20260619 10:30:04.586561 31897 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.124:33253
I20260619 10:30:04.586609 31964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.124:33253 every 8 connection(s)
I20260619 10:30:04.586957 31897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
I20260619 10:30:04.587584 31965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:30:04.588973 31965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:04.590852 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 31897
I20260619 10:30:04.590942 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781864754199919-9577-0/minicluster-data/master-2/wal/instance
I20260619 10:30:04.591123 9577 external_mini_cluster.cc:1428] Setting key a7de8aa2b096e6371d0693101954aa0c
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
I20260619 10:30:04.596266 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.586550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50548)
0619 10:30:04.586623 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.586635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:04.586689 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:04.588076 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:04.588077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:04.588233 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:04.588274 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.588276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.589352 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.589353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.590127 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.590129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.590841 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.590843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.590903 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.591596 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:04.591601 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:04.592939 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:04.594792 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.594793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.594793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.594990 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.595495 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.595496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.595496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.595563 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.595955 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:04.595956 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:04.596003 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 10:30:04.596064 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:30:04.596169 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 10:30:04.596491 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.586885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.9.90.124:33253)
0619 10:30:04.587427 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.587435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.587462 (+ 27us) server_negotiation.cc:407] Connection header received
0619 10:30:04.587897 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.587904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.587949 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.588074 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.588374 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.589270 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.590211 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.590758 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.590858 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.593024 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.593030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.593030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.593036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.594716 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.595080 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.595083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.595084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.595416 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.595645 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.595646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.595646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.595878 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.595947 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.596135 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.596286 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":470,"threads_started":1}
I20260619 10:30:04.597234 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.589563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.9.90.126:40493)
0619 10:30:04.589611 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.589612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.589697 (+ 85us) server_negotiation.cc:407] Connection header received
0619 10:30:04.589768 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.589769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.589776 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.589865 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.590351 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.591006 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.592003 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.592264 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.592329 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.594525 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.594526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.594527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.594533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.595919 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.596340 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.596341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.596342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.596396 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.596721 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.596722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.596722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.596822 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.596880 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.597071 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.597140 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:30:04.597237 31968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.589469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:40493 (local address 127.0.0.1:41256)
0619 10:30:04.589609 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.589649 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:04.589712 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:04.589885 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:04.589888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:04.590198 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:04.590273 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.590275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.591089 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.591089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.591935 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.591937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.592382 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.592383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.592460 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.592898 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:04.592906 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:04.594427 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:04.596004 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.596011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.596013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.596276 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.596455 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.596456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.596456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.596512 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.596894 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:04.596900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:04.596945 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 10:30:04.597014 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 10:30:04.597136 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260619 10:30:04.597963 31965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:04.598278 31890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69: Bootstrap starting.
I20260619 10:30:04.599175 31890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69: Neither blocks nor log segments found. Creating new log.
I20260619 10:30:04.599408 31890 log.cc:824] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69: Log is configured to *not* fsync() on all Append() calls
I20260619 10:30:04.600179 31890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69: No bootstrap required, opened a new log
I20260619 10:30:04.600692 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.591464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.9.90.124:33253)
0619 10:30:04.591748 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.591750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.591756 (+ 6us) server_negotiation.cc:407] Connection header received
0619 10:30:04.591771 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.591772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.591780 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.591835 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.592483 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.593123 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.594002 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.594245 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.594300 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.597060 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.597061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.597061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.597067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.599525 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.599893 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.599894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.599894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.599947 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.600120 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.600120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.600121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.600235 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.600353 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.600537 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.600603 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":59,"threads_started":1}
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
I20260619 10:30:04.602202 31890 raft_consensus.cc:348] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } }
I20260619 10:30:04.602335 31890 raft_consensus.cc:729] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af907bfb06954d10b9710b6ae0cf1b69, State: Initialized, Role: FOLLOWER
I20260619 10:30:04.602425 31890 consensus_queue.cc:260] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } }
I20260619 10:30:04.602730 31890 sys_catalog.cc:564] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:30:04.602870 31972 sys_catalog.cc:455] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.602952 31972 sys_catalog.cc:458] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 10:30:04.604348 31983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:30:04.604440 31983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:30:04.604723 31968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.598195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46437 (local address 127.0.0.1:56990)
0619 10:30:04.598245 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.598255 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:04.598296 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:04.598407 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:04.598407 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:04.598543 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:04.598590 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.598592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.599398 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.599399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.600059 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.600062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.600502 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.600503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.600569 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.600953 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:04.600957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:04.602258 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:04.603769 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.603769 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.603770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.603918 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.604100 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.604101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.604101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.604167 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.604467 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:04.604468 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:04.604507 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 10:30:04.604564 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 10:30:04.604638 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 10:30:04.605235 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.598262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.9.90.125:46437)
0619 10:30:04.598299 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.598301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.598327 (+ 26us) server_negotiation.cc:407] Connection header received
0619 10:30:04.598346 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.598347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.598356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.598404 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.598679 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.599301 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.600162 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.600424 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.600486 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.602331 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.602332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.602333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.602339 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.603706 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.603990 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.603991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.603991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.604039 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.604232 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.604233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.604234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.604385 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.604442 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.605071 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.605152 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:30:04.605603 31965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:04.604799 31822 master_runner.cc:186] Error getting master registration for 127.9.90.126:40493: OK, instance_id { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" instance_seqno: 1781865004297163 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:30:04.606994 31822 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:30:04.611996 31968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.605817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50562)
0619 10:30:04.605851 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.605859 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:04.605891 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:04.606013 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:04.606014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:04.606126 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:04.606148 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.606149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.606703 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.606704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.607365 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.607367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.607700 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.607700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.607742 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.608071 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:04.608075 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:04.609366 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:04.610892 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.610892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.610893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.611073 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.611301 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.611301 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.611302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.611365 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.611689 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:04.611690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:04.611729 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 10:30:04.611802 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 10:30:04.611876 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 10:30:04.612068 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.605863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.9.90.124:33253)
0619 10:30:04.605940 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.605941 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.605946 (+ 5us) server_negotiation.cc:407] Connection header received
0619 10:30:04.605956 (+ 10us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.605956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.605962 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.605997 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.606212 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.606638 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.607434 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.607641 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.607690 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.609439 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.609440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.609440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.609446 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.610818 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.611162 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.611163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.611163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.611223 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.611434 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.611435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.611435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.611601 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.611681 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.611843 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.611935 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":66}
I20260619 10:30:04.613205 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.601395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.9.90.126:40493)
0619 10:30:04.603957 (+ 2562us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.603959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.603968 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:30:04.603987 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.603988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.603997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.604055 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.604581 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.605246 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.606716 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.606993 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.607048 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.610157 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.610159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.610159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.610166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.612002 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.612345 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.612346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.612347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.612398 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.612567 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.612568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.612568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.612703 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.612868 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.612967 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.613120 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2542}
I20260619 10:30:04.614019 31965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f: Bootstrap starting.
I20260619 10:30:04.614648 31965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f: Neither blocks nor log segments found. Creating new log.
I20260619 10:30:04.614889 31965 log.cc:824] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f: Log is configured to *not* fsync() on all Append() calls
I20260619 10:30:04.615984 31965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f: No bootstrap required, opened a new log
I20260619 10:30:04.617406 31965 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } }
I20260619 10:30:04.617520 31965 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd54abf834984375b113d7c245ebe84f, State: Initialized, Role: FOLLOWER
I20260619 10:30:04.617594 31965 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } }
I20260619 10:30:04.617893 31965 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:30:04.617853 31986 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.618021 31986 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
W20260619 10:30:04.619815 31997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:30:04.619894 31997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:30:04.621507 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.613718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.9.90.125:46437)
0619 10:30:04.613754 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.613756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.613763 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:30:04.613781 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.613782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.613811 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.613862 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.614360 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.615075 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.615757 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.616035 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.616098 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.618640 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.618641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.618642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.618648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.620165 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.620509 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.620510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.620511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.620559 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.620766 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.620767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.620767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.620884 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.621016 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.621348 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.621413 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:30:04.619684 31897 master_runner.cc:186] Error getting master registration for 127.9.90.126:40493: OK, instance_id { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" instance_seqno: 1781865004297163 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:30:04.621765 31897 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:30:04.622931 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:30:04.624089 31789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a7bb84051e57447b9eaa8b69c5a31888"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41270
I20260619 10:30:04.637985 31815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } attempt: 2
Jun 19 10:30:04 dist-test-slave-hw7q krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
I20260619 10:30:04.645817 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.638246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:33253 (local address 127.0.0.1:50574)
0619 10:30:04.638292 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:04.638302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:04.638337 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:04.638481 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:04.638482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:04.638646 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:04.638727 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.638729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.639618 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.639619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.640424 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:04.640427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.640954 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.640955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:04.641014 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.641522 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:04.641527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:04.642991 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:04.644710 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.644712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.644712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.644919 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.645131 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:04.645132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:04.645132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:04.645191 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:04.645475 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:04.645476 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:04.645531 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 10:30:04.645601 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 10:30:04.645693 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:30:04.645846 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.638310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.9.90.124:33253)
0619 10:30:04.638362 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.638363 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.638372 (+ 9us) server_negotiation.cc:407] Connection header received
0619 10:30:04.638394 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.638395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.638404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.638464 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.638787 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.639537 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.640517 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.640842 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.640899 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.643084 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.643086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.643086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.643093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.644623 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.644995 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.644996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.644996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.645068 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.645267 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.645269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.645269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.645410 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.645471 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.645638 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.645723 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:30:04.647905 31815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888: Bootstrap starting.
I20260619 10:30:04.648625 31815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888: Neither blocks nor log segments found. Creating new log.
I20260619 10:30:04.648933 31815 log.cc:824] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888: Log is configured to *not* fsync() on all Append() calls
I20260619 10:30:04.649907 31815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888: No bootstrap required, opened a new log
I20260619 10:30:04.651225 31815 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } }
I20260619 10:30:04.651367 31815 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7bb84051e57447b9eaa8b69c5a31888, State: Initialized, Role: FOLLOWER
I20260619 10:30:04.651448 31815 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } }
I20260619 10:30:04.651752 32000 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.651798 31789 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 10:30:04.651768 31815 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:30:04.651976 31789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:30:04.651937 32000 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:04.652752 31789 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } }
I20260619 10:30:04.652994 31789 leader_election.cc:288] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [CANDIDATE]: Term 1 election: Requested vote from peers af907bfb06954d10b9710b6ae0cf1b69 (127.9.90.125:46437), cd54abf834984375b113d7c245ebe84f (127.9.90.124:33253)
I20260619 10:30:04.653340 31864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7bb84051e57447b9eaa8b69c5a31888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af907bfb06954d10b9710b6ae0cf1b69"
I20260619 10:30:04.653469 31864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:30:04.654210 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7bb84051e57447b9eaa8b69c5a31888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd54abf834984375b113d7c245ebe84f"
I20260619 10:30:04.654299 31864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7bb84051e57447b9eaa8b69c5a31888 in term 1.
I20260619 10:30:04.654337 31939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:30:04.654557 31766 leader_election.cc:302] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7bb84051e57447b9eaa8b69c5a31888, af907bfb06954d10b9710b6ae0cf1b69; no voters:
I20260619 10:30:04.654698 32000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:30:04.654819 32000 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [term 1 LEADER]: Becoming Leader. State: Replica: a7bb84051e57447b9eaa8b69c5a31888, State: Running, Role: LEADER
I20260619 10:30:04.654906 32000 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } }
I20260619 10:30:04.655138 31939 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7bb84051e57447b9eaa8b69c5a31888 in term 1.
I20260619 10:30:04.655288 32000 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7bb84051e57447b9eaa8b69c5a31888. Latest consensus state: current_term: 1 leader_uuid: "a7bb84051e57447b9eaa8b69c5a31888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.655357 32000 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [sys.catalog]: This master's current role is: LEADER
W20260619 10:30:04.657492 32019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:30:04.657543 32019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:30:04.661886 32008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:30:04.662062 32008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:30:04.662361 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.654186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.9.90.126:40493)
0619 10:30:04.654436 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.654438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.654539 (+ 101us) server_negotiation.cc:407] Connection header received
0619 10:30:04.654705 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.654707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.654718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.654751 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.655807 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.656539 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.657551 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.657858 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.657895 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.659422 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.659423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.659424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.659430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.661082 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.661467 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.661468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.661469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.661519 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.661699 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.661700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.661700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.661833 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.661899 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.662217 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.662287 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260619 10:30:04.669566 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.662394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.9.90.124:33253)
0619 10:30:04.662430 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.662432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.663112 (+ 680us) server_negotiation.cc:407] Connection header received
0619 10:30:04.663212 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.663214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.663222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.663249 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.663731 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.664419 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.665358 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.665642 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.665678 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.666957 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.666959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.666959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.666965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.668451 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.668813 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.668814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.668815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.668865 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.669038 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.669038 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.669039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.669162 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.669227 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.669411 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.669478 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:30:04.674829 31864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [term 1 FOLLOWER]: Refusing update from remote peer a7bb84051e57447b9eaa8b69c5a31888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:30:04.675266 31939 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [term 1 FOLLOWER]: Refusing update from remote peer a7bb84051e57447b9eaa8b69c5a31888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:30:04.675550 32000 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:30:04.675904 32000 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:30:04.677155 31986 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7bb84051e57447b9eaa8b69c5a31888. Latest consensus state: current_term: 1 leader_uuid: "a7bb84051e57447b9eaa8b69c5a31888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.677243 31986 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:04.677533 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:04.656762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.9.90.125:46437)
0619 10:30:04.656934 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.656935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.657691 (+ 756us) server_negotiation.cc:407] Connection header received
0619 10:30:04.657781 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.657782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.657818 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.657845 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.658534 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.659594 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.660587 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:04.660896 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:04.660935 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:04.673888 (+ 12953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:04.673890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:04.673890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:04.673898 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:04.675839 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.676312 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.676313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.676313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.676368 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:04.676621 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:04.676622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:04.676622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:04.676787 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:04.677323 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:04.677364 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 10:30:04.677437 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260619 10:30:04.678120 31972 sys_catalog.cc:455] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7bb84051e57447b9eaa8b69c5a31888. Latest consensus state: current_term: 1 leader_uuid: "a7bb84051e57447b9eaa8b69c5a31888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.678225 31972 sys_catalog.cc:458] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:04.679338 32000 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7bb84051e57447b9eaa8b69c5a31888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.679422 32000 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:04.679491 32023 mvcc.cc:205] Tried to move back new op lower bound from 7298519059145154560 to 7298519059067138048. Current Snapshot: MvccSnapshot[applied={T|T < 7298519059145154560}]
I20260619 10:30:04.679857 31972 sys_catalog.cc:455] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7bb84051e57447b9eaa8b69c5a31888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.679939 31972 sys_catalog.cc:458] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:04.680537 32008 catalog_manager.cc:1347] Generated new cluster ID: 8a54efbc92564405943da908819822b0
I20260619 10:30:04.680585 32008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:30:04.686551 32006 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7bb84051e57447b9eaa8b69c5a31888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.686651 32006 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888 [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:04.686964 31986 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7bb84051e57447b9eaa8b69c5a31888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bb84051e57447b9eaa8b69c5a31888" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 40493 } } peers { permanent_uuid: "af907bfb06954d10b9710b6ae0cf1b69" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46437 } } peers { permanent_uuid: "cd54abf834984375b113d7c245ebe84f" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 33253 } } }
I20260619 10:30:04.687062 31986 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:04.687256 32024 mvcc.cc:205] Tried to move back new op lower bound from 7298519059145154560 to 7298519059067138048. Current Snapshot: MvccSnapshot[applied={T|T < 7298519059145154560}]
I20260619 10:30:04.693058 32026 mvcc.cc:205] Tried to move back new op lower bound from 7298519059145154560 to 7298519059067138048. Current Snapshot: MvccSnapshot[applied={T|T < 7298519059145154560}]
I20260619 10:30:04.705247 32008 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:30:04.705978 32008 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:30:04.715791 32008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7bb84051e57447b9eaa8b69c5a31888: Generated new TSK 0
I20260619 10:30:04.716845 32008 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:30:04.740136 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.738639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.9.90.125:46437)
0619 10:30:04.738684 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.738686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.739543 (+ 857us) server_negotiation.cc:407] Connection header received
0619 10:30:04.739634 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.739635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.739645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.739674 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.740066 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57016: BlockingRecv error: recv got EOF from 127.0.0.1:57016 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 10:30:04.740852 31818 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.738993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.9.90.126:40493)
0619 10:30:04.739031 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.739033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.740368 (+ 1335us) server_negotiation.cc:407] Connection header received
0619 10:30:04.740454 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.740455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.740464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.740489 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.740794 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41276: BlockingRecv error: recv got EOF from 127.0.0.1:41276 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 10:30:04.741524 31970 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.739292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.9.90.124:33253)
0619 10:30:04.739332 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.739335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.741056 (+ 1721us) server_negotiation.cc:407] Connection header received
0619 10:30:04.741142 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.741143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.741164 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.741193 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.741470 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50594: BlockingRecv error: recv got EOF from 127.0.0.1:50594 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:30:04.743707 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.743051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.9.90.125:46437)
0619 10:30:04.743082 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.743083 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.743258 (+ 175us) server_negotiation.cc:407] Connection header received
0619 10:30:04.743348 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.743348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.743366 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.743391 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.743663 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57026: BlockingRecv error: recv got EOF from 127.0.0.1:57026 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260619 10:30:04.746688 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.745914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.9.90.125:46437)
0619 10:30:04.745945 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.745947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.746202 (+ 255us) server_negotiation.cc:407] Connection header received
0619 10:30:04.746296 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.746297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.746306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.746333 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.746646 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57034: BlockingRecv error: recv got EOF from 127.0.0.1:57034 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 10:30:04.750417 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.749681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.9.90.125:46437)
0619 10:30:04.749714 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.749715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.749957 (+ 242us) server_negotiation.cc:407] Connection header received
0619 10:30:04.750044 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.750045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.750054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.750080 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.750374 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57038: BlockingRecv error: recv got EOF from 127.0.0.1:57038 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 10:30:04.756358 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.755522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.9.90.125:46437)
0619 10:30:04.755556 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.755557 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.755814 (+ 257us) server_negotiation.cc:407] Connection header received
0619 10:30:04.755914 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.755915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.755925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.755952 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.756310 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57052: BlockingRecv error: recv got EOF from 127.0.0.1:57052 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 10:30:04.766675 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.765668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.9.90.125:46437)
0619 10:30:04.765715 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.765717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.766052 (+ 335us) server_negotiation.cc:407] Connection header received
0619 10:30:04.766168 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.766170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.766182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.766214 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.766613 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57064: BlockingRecv error: recv got EOF from 127.0.0.1:57064 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 10:30:04.787724 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.786756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.9.90.125:46437)
0619 10:30:04.786805 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.786807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.787099 (+ 292us) server_negotiation.cc:407] Connection header received
0619 10:30:04.787250 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.787251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.787264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.787294 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.787660 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57068: BlockingRecv error: recv got EOF from 127.0.0.1:57068 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260619 10:30:04.821512 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.820925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.9.90.125:46437)
0619 10:30:04.820986 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.820988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.821072 (+ 84us) server_negotiation.cc:407] Connection header received
0619 10:30:04.821119 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.821119 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.821130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.821191 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.821464 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57076: BlockingRecv error: recv got EOF from 127.0.0.1:57076 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 10:30:04.887290 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:04.886680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.9.90.125:46437)
0619 10:30:04.886740 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:04.886742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:04.886820 (+ 78us) server_negotiation.cc:407] Connection header received
0619 10:30:04.886870 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:04.886870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:04.886880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:04.886940 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:04.887241 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57092: BlockingRecv error: recv got EOF from 127.0.0.1:57092 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 10:30:05.017035 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:05.016431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.9.90.125:46437)
0619 10:30:05.016500 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:05.016502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:05.016569 (+ 67us) server_negotiation.cc:407] Connection header received
0619 10:30:05.016629 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:05.016629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:05.016639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:05.016691 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:05.016970 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57102: BlockingRecv error: recv got EOF from 127.0.0.1:57102 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 10:30:05.274869 31894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:05.274182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.9.90.125:46437)
0619 10:30:05.274236 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:05.274238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:05.274280 (+ 42us) server_negotiation.cc:407] Connection header received
0619 10:30:05.274351 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:05.274352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:05.274362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:05.274438 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:05.274814 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57116: BlockingRecv error: recv got EOF from 127.0.0.1:57116 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 10:30:05.604914 31983 catalog_manager.cc:1259] Loaded cluster ID: 8a54efbc92564405943da908819822b0
I20260619 10:30:05.604985 31983 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69: loading cluster ID for follower catalog manager: success
I20260619 10:30:05.605772 31983 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69: acquiring CA information for follower catalog manager: success
I20260619 10:30:05.606119 31983 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af907bfb06954d10b9710b6ae0cf1b69: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:30:05.620353 31997 catalog_manager.cc:1259] Loaded cluster ID: 8a54efbc92564405943da908819822b0
I20260619 10:30:05.620432 31997 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f: loading cluster ID for follower catalog manager: success
I20260619 10:30:05.621327 31997 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f: acquiring CA information for follower catalog manager: success
I20260619 10:30:05.621608 31997 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd54abf834984375b113d7c245ebe84f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:30:05.790897 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:05.788111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.9.90.125:46437)
0619 10:30:05.788237 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:05.788239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:05.788249 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:30:05.788313 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:05.788314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:05.788363 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:05.788437 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:05.788694 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.789194 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.790078 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.790276 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.790332 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:05.790547 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:05.790666 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:05.790737 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:05.790786 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 10:30:05.790809 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260619 10:30:05.795279 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:05.792867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.9.90.124:33253)
0619 10:30:05.792982 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:05.792984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:05.792992 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:30:05.793029 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:05.793030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:05.793064 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:05.793133 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:05.793264 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.793764 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.794533 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.794736 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.794789 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:05.794943 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:05.795047 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:05.795113 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:05.795169 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 10:30:05.795191 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":47,"threads_started":1}
I20260619 10:30:05.799714 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:05.796976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.9.90.124:33253)
0619 10:30:05.797012 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:05.797013 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:05.797148 (+ 135us) server_negotiation.cc:407] Connection header received
0619 10:30:05.797185 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:05.797186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:05.797195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:05.797236 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:05.797391 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.797916 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.798933 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.799183 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.799235 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:05.799377 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:05.799408 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:05.799462 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:05.799614 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 10:30:05.799645 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:30:05.799803 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:05.797075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.9.90.125:46437)
0619 10:30:05.797111 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:05.797112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:05.797120 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:30:05.797173 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:05.797173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:05.797182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:05.797233 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:05.797339 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.798072 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.799067 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.799337 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.799390 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:05.799534 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:05.799562 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:05.799603 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:05.799707 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 10:30:05.799728 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 10:30:05.799852 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:05.797021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.9.90.126:40493)
0619 10:30:05.797122 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:05.797124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:05.797132 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:30:05.797174 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:05.797174 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:05.797219 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:05.797293 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:05.797435 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.798180 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.798925 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:05.799189 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:05.799244 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:05.799457 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:05.799559 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:05.799610 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:05.799680 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 10:30:05.799703 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
I20260619 10:30:05.801447 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 31747
I20260619 10:30:05.806028 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 31822
I20260619 10:30:05.810269 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 31897
2026-06-19T10:30:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2687 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 10:30:05 dist-test-slave-hw7q krb5kdc[32112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 10:30:05 dist-test-slave-hw7q krb5kdc[32112](info): set up 2 sockets
Jun 19 10:30:05 dist-test-slave-hw7q krb5kdc[32112](info): commencing operation
krb5kdc: starting...
W20260619 10:30:06.757719 9577 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/test-user.keytab.
Jun 19 10:30:06 dist-test-slave-hw7q krb5kdc[32112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T10:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T10:30:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
Entry for principal kudu/127.9.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab.
I20260619 10:30:06.866513 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36985
--webserver_interface=127.9.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:36985,127.9.90.125:46253,127.9.90.124:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 10:30:06.944865 32125 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:30:06.945075 32125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:30:06.945097 32125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:30:06.945122 32125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:30:06.946257 32125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:30:06.946296 32125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:30:06.946313 32125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:30:06.946329 32125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:30:06.946344 32125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:30:06.947794 32125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:36985,127.9.90.125:46253,127.9.90.124:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.126.keytab
--principal=kudu/127.9.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.126:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:30:06.948033 32125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:30:06.948256 32125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 10:30:06.953122 32125 server_base.cc:1034] running on GCE node
W20260619 10:30:06.953061 32133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:06.953074 32131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:06.953022 32130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:06.953616 32125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:30:06.953948 32125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:30:06.955089 32125 hybrid_clock.cc:648] HybridClock initialized: now 1781865006955068 us; error 40 us; skew 500 ppm
Jun 19 10:30:06 dist-test-slave-hw7q krb5kdc[32112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865006, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:30:06.957402 32125 init.cc:377] Logged in from keytab as kudu/127.9.90.126@KRBTEST.COM (short username kudu)
I20260619 10:30:06.957890 32125 webserver.cc:460] Webserver started at http://127.9.90.126:35527/ using document root <none> and password file <none>
I20260619 10:30:06.958120 32125 fs_manager.cc:359] Metadata directory not provided
I20260619 10:30:06.958175 32125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:30:06.958302 32125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:30:06.959312 32125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/data/instance:
uuid: "a64ec77ba795429c947f09f94c8f654e"
format_stamp: "Formatted at 2026-06-19 10:30:06 on dist-test-slave-hw7q"
server_key: "4278e4b22b40c3562593e47e5d2b1c68"
server_key_iv: "906dcc39e56d2a5471e0caf62e427211"
server_key_version: "encryptionkey@0"
I20260619 10:30:06.959643 32125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/wal/instance:
uuid: "a64ec77ba795429c947f09f94c8f654e"
format_stamp: "Formatted at 2026-06-19 10:30:06 on dist-test-slave-hw7q"
server_key: "4278e4b22b40c3562593e47e5d2b1c68"
server_key_iv: "906dcc39e56d2a5471e0caf62e427211"
server_key_version: "encryptionkey@0"
I20260619 10:30:06.960877 32125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 10:30:06.961513 32140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:30:06.961681 32125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 10:30:06.961761 32125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/wal
uuid: "a64ec77ba795429c947f09f94c8f654e"
format_stamp: "Formatted at 2026-06-19 10:30:06 on dist-test-slave-hw7q"
server_key: "4278e4b22b40c3562593e47e5d2b1c68"
server_key_iv: "906dcc39e56d2a5471e0caf62e427211"
server_key_version: "encryptionkey@0"
I20260619 10:30:06.961854 32125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:30:06.988698 32125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:30:06.990461 32125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:30:06.990600 32125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:30:06.994275 32125 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.126:36985
I20260619 10:30:06.994314 32192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.126:36985 every 8 connection(s)
I20260619 10:30:06.994638 32125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/data/info.pb
I20260619 10:30:06.995548 32193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:30:06.997040 32193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:07.000896 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 32125
I20260619 10:30:07.000980 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-0/wal/instance
I20260619 10:30:07.001133 9577 external_mini_cluster.cc:1428] Setting key 6852ce98016ae97c0fb9ce5477013642
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865006, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:30:07.007660 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:06.997767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.9.90.126:36985)
0619 10:30:06.997901 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:06.997911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:06.997999 (+ 88us) server_negotiation.cc:407] Connection header received
0619 10:30:06.998542 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:06.998544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:06.998559 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:06.998633 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:06.999163 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:06.999951 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.000831 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.001247 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.001337 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.003772 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.003779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.003780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.003786 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.005500 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.006042 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.006046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.006047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.006181 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.006448 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.006449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.006450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.006784 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.006864 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.007185 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.007369 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":107,"thread_start_us":92,"threads_started":1}
I20260619 10:30:07.007675 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:06.997687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36985 (local address 127.0.0.1:48192)
0619 10:30:06.997875 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:06.997918 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:06.998388 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:06.998657 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:06.998659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:06.998917 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:06.999060 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:06.999065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.000045 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.000046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.000737 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.000739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.001357 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.001358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.001431 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.001972 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:07.001982 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:07.003645 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:07.005608 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.005615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.005618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.005941 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.006259 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.006260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.006261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.006354 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.006858 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:07.006866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:07.007006 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 10:30:07.007130 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 10:30:07.007451 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":75,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.9.90.125@KRBTEST.COM; defaulting to no policy
I20260619 10:30:07.009142 32193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:07.009663 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.009365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46253 (local address 127.0.0.1:57588)
0619 10:30:07.009512 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.009635 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260619 10:30:07.010052 32141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
I20260619 10:30:07.010517 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.010429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46253 (local address 127.0.0.1:57592)
0619 10:30:07.010463 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.010493 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260619 10:30:07.010672 32193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.125:46253: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab.
Entry for principal kudu/127.9.90.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab.
I20260619 10:30:07.026422 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.125:46253
--webserver_interface=127.9.90.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab
--principal=kudu/127.9.90.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:36985,127.9.90.125:46253,127.9.90.124:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:30:07.059899 32193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } attempt: 1
I20260619 10:30:07.060410 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.060237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46253 (local address 127.0.0.1:57606)
0619 10:30:07.060311 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.060367 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 10:30:07.060797 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.060685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46253 (local address 127.0.0.1:57608)
0619 10:30:07.060738 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.060772 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 10:30:07.060902 32193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.125:46253: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
W20260619 10:30:07.103827 32200 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:30:07.104033 32200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:30:07.104048 32200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:30:07.104067 32200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:30:07.105188 32200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:30:07.105228 32200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:30:07.105242 32200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:30:07.105252 32200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:30:07.105263 32200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:30:07.106714 32200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:36985,127.9.90.125:46253,127.9.90.124:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.125.keytab
--principal=kudu/127.9.90.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.125:46253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.90.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:30:07.106932 32200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:30:07.107146 32200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:30:07.111856 32205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:07.111829 32206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:07.111809 32208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:07.112026 32200 server_base.cc:1034] running on GCE node
I20260619 10:30:07.112238 32200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:30:07.112509 32200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:30:07.113654 32200 hybrid_clock.cc:648] HybridClock initialized: now 1781865007113633 us; error 34 us; skew 500 ppm
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:30:07.116025 32200 init.cc:377] Logged in from keytab as kudu/127.9.90.125@KRBTEST.COM (short username kudu)
I20260619 10:30:07.116452 32200 webserver.cc:460] Webserver started at http://127.9.90.125:40839/ using document root <none> and password file <none>
I20260619 10:30:07.116652 32200 fs_manager.cc:359] Metadata directory not provided
I20260619 10:30:07.116693 32200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:30:07.116811 32200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:30:07.117885 32200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/data/instance:
uuid: "3e34e80981cb4a48ade0ca7966739a07"
format_stamp: "Formatted at 2026-06-19 10:30:07 on dist-test-slave-hw7q"
server_key: "6bd0dac04445edcc47f8c3e439454d39"
server_key_iv: "f6e1d81c730986e0027e67b2088cf92c"
server_key_version: "encryptionkey@0"
I20260619 10:30:07.118225 32200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/wal/instance:
uuid: "3e34e80981cb4a48ade0ca7966739a07"
format_stamp: "Formatted at 2026-06-19 10:30:07 on dist-test-slave-hw7q"
server_key: "6bd0dac04445edcc47f8c3e439454d39"
server_key_iv: "f6e1d81c730986e0027e67b2088cf92c"
server_key_version: "encryptionkey@0"
I20260619 10:30:07.119434 32200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:30:07.120251 32215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:30:07.120406 32200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 10:30:07.120482 32200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/wal
uuid: "3e34e80981cb4a48ade0ca7966739a07"
format_stamp: "Formatted at 2026-06-19 10:30:07 on dist-test-slave-hw7q"
server_key: "6bd0dac04445edcc47f8c3e439454d39"
server_key_iv: "f6e1d81c730986e0027e67b2088cf92c"
server_key_version: "encryptionkey@0"
I20260619 10:30:07.120550 32200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:30:07.128954 32200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:30:07.129060 32193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } attempt: 2
I20260619 10:30:07.129438 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.129321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46253 (local address 127.0.0.1:57620)
0619 10:30:07.129372 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.129404 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 10:30:07.129753 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.129663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46253 (local address 127.0.0.1:57630)
0619 10:30:07.129706 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.129729 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 10:30:07.129858 32193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.125:46253: Network error: Client connection negotiation failed: client connection to 127.9.90.125:46253: connect: Connection refused (error 111)
I20260619 10:30:07.131131 32200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:30:07.131270 32200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:30:07.134819 32200 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.125:46253
I20260619 10:30:07.134847 32267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.125:46253 every 8 connection(s)
I20260619 10:30:07.135171 32200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/data/info.pb
I20260619 10:30:07.136082 32268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:30:07.137560 32268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:07.140569 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 32200
I20260619 10:30:07.140657 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-1/wal/instance
I20260619 10:30:07.140836 9577 external_mini_cluster.cc:1428] Setting key 41faf0ea6e6fc7e66dd2e9ce136f6713
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:30:07.146451 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.138275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.9.90.126:36985)
0619 10:30:07.138341 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.138342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.138537 (+ 195us) server_negotiation.cc:407] Connection header received
0619 10:30:07.138975 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.138977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.138986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.139042 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.139576 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.140051 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.141056 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.141327 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.141402 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.143645 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.143647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.143648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.143656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.145126 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.145505 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.145506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.145507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.145569 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.145817 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.145817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.145818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.145958 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.146025 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.146276 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.146360 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 10:30:07.146497 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.138229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36985 (local address 127.0.0.1:48200)
0619 10:30:07.138382 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.138423 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:07.138880 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:07.139094 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:07.139097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:07.139325 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:07.139509 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.139513 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.140115 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.140116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.140974 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.140977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.141420 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.141421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.141509 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.142015 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:07.142024 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:07.143504 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:07.145216 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.145222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.145224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.145432 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.145639 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.145640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.145640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.145726 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.146027 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:07.146031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:07.146151 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 10:30:07.146229 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 10:30:07.146342 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260619 10:30:07.147094 32268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.90.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.90.124@KRBTEST.COM" created.
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
I20260619 10:30:07.153869 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.147320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46253 (local address 127.0.0.1:57640)
0619 10:30:07.147489 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.147500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:07.147532 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:07.147650 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:07.147651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:07.147772 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:07.147812 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.147813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.148650 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.148651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.149128 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.149130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.149654 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.149654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.149714 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.150088 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:07.150093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:07.151433 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:07.152974 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.152975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.152975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.153118 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.153317 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.153318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.153318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.153363 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.153622 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:07.153623 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:07.153657 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 10:30:07.153702 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 10:30:07.153770 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260619 10:30:07.153983 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.147376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.9.90.125:46253)
0619 10:30:07.147510 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.147515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.147547 (+ 32us) server_negotiation.cc:407] Connection header received
0619 10:30:07.147567 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.147569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.147582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.147635 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.147877 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.148572 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.149199 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.149580 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.149641 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.151514 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.151521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.151522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.151530 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.152898 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.153197 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.153200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.153200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.153257 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.153415 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.153416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.153416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.153566 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.153639 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.153734 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.153865 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260619 10:30:07.154526 32268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:07.154839 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.154737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42326)
0619 10:30:07.154780 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.154821 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 10:30:07.154973 32217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
I20260619 10:30:07.155347 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.155287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42334)
0619 10:30:07.155311 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.155332 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 10:30:07.155447 32268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:42017: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.90.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab.
Entry for principal kudu/127.9.90.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab.
I20260619 10:30:07.165045 9577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzGZPou/build/release/bin/kudu
/tmp/dist-test-taskzGZPou/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.124:42017
--webserver_interface=127.9.90.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab
--principal=kudu/127.9.90.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.90.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.126:36985,127.9.90.125:46253,127.9.90.124:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 10:30:07.204703 32268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } attempt: 1
I20260619 10:30:07.205214 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.205081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42342)
0619 10:30:07.205148 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.205182 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 10:30:07.205605 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.205472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42350)
0619 10:30:07.205530 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.205573 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 10:30:07.205721 32268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:42017: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
I20260619 10:30:07.220999 32193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } attempt: 3
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865006, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
I20260619 10:30:07.228679 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.221393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46253 (local address 127.0.0.1:57654)
0619 10:30:07.221463 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.221477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:07.221537 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:07.221675 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:07.221675 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:07.221857 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:07.221924 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.221925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.222839 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.222840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.223382 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.223385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.223922 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.223923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.223996 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.224445 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:07.224450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:07.225901 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:07.227723 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.227724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.227724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.227881 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.228086 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.228086 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.228087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.228132 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.228379 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:07.228379 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:07.228419 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 10:30:07.228465 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 10:30:07.228578 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 10:30:07.228680 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.221461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.9.90.125:46253)
0619 10:30:07.221540 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.221543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.221554 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:30:07.221582 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.221583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.221597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.221662 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.221995 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.222756 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.223472 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.223837 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.223906 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.225993 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.225995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.225995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.226004 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.227641 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.227956 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.227957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.227958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.228024 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.228195 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.228195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.228196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.228327 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.228389 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.228495 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.228585 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 10:30:07.229228 32193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:07.229539 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.229441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42366)
0619 10:30:07.229486 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.229514 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:30:07.229884 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.229809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42370)
0619 10:30:07.229839 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.229864 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260619 10:30:07.229969 32193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:42017: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
W20260619 10:30:07.246636 32275 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 10:30:07.246836 32275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 10:30:07.246851 32275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 10:30:07.246872 32275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 10:30:07.248034 32275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 10:30:07.248080 32275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 10:30:07.248093 32275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 10:30:07.248104 32275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 10:30:07.248114 32275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 10:30:07.249747 32275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.126:36985,127.9.90.125:46253,127.9.90.124:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/krb5kdc/kudu_127.9.90.124.keytab
--principal=kudu/127.9.90.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.124:42017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.90.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 09:32:44 UTC on ccycloud.quasar-gbtldp.root.comops.site
build id 4455440
I20260619 10:30:07.250047 32275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 10:30:07.250272 32275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 10:30:07.254859 32280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:07.254901 32283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 10:30:07.254878 32281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 10:30:07.255055 32275 server_base.cc:1034] running on GCE node
I20260619 10:30:07.255306 32275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 10:30:07.255565 32275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 10:30:07.256704 32275 hybrid_clock.cc:648] HybridClock initialized: now 1781865007256691 us; error 30 us; skew 500 ppm
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 10:30:07.259137 32275 init.cc:377] Logged in from keytab as kudu/127.9.90.124@KRBTEST.COM (short username kudu)
I20260619 10:30:07.259567 32275 webserver.cc:460] Webserver started at http://127.9.90.124:33161/ using document root <none> and password file <none>
I20260619 10:30:07.259788 32275 fs_manager.cc:359] Metadata directory not provided
I20260619 10:30:07.259837 32275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 10:30:07.259959 32275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 10:30:07.261116 32193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } attempt: 1
I20260619 10:30:07.261037 32275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/data/instance:
uuid: "3165c9104a6f48128a1dbc3af5d7ef07"
format_stamp: "Formatted at 2026-06-19 10:30:07 on dist-test-slave-hw7q"
server_key: "af5b1d7fcee8e832af9ce4fd8c7ebd1d"
server_key_iv: "c1461554aac94f295b3202655455cf8a"
server_key_version: "encryptionkey@0"
I20260619 10:30:07.261421 32275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/wal/instance:
uuid: "3165c9104a6f48128a1dbc3af5d7ef07"
format_stamp: "Formatted at 2026-06-19 10:30:07 on dist-test-slave-hw7q"
server_key: "af5b1d7fcee8e832af9ce4fd8c7ebd1d"
server_key_iv: "c1461554aac94f295b3202655455cf8a"
server_key_version: "encryptionkey@0"
I20260619 10:30:07.261570 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.261413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42380)
0619 10:30:07.261489 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.261528 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 10:30:07.261929 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.261848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42396)
0619 10:30:07.261880 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.261905 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260619 10:30:07.262038 32193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:42017: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
I20260619 10:30:07.262912 32275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 10:30:07.263628 32290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 10:30:07.263805 32275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 10:30:07.263886 32275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/data,/tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/wal
uuid: "3165c9104a6f48128a1dbc3af5d7ef07"
format_stamp: "Formatted at 2026-06-19 10:30:07 on dist-test-slave-hw7q"
server_key: "af5b1d7fcee8e832af9ce4fd8c7ebd1d"
server_key_iv: "c1461554aac94f295b3202655455cf8a"
server_key_version: "encryptionkey@0"
I20260619 10:30:07.263957 32275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 10:30:07.273871 32268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } attempt: 2
I20260619 10:30:07.274258 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.274148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42400)
0619 10:30:07.274204 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.274233 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 10:30:07.274571 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.274487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42414)
0619 10:30:07.274525 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.274548 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 10:30:07.274662 32268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.90.124:42017: Network error: Client connection negotiation failed: client connection to 127.9.90.124:42017: connect: Connection refused (error 111)
I20260619 10:30:07.283285 32275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 10:30:07.285388 32275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 10:30:07.285522 32275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 10:30:07.289124 32275 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.124:42017
I20260619 10:30:07.289178 32342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.90.124:42017 every 8 connection(s)
I20260619 10:30:07.289480 32275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/data/info.pb
I20260619 10:30:07.290472 32343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 10:30:07.291931 32343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:30:07.299602 9577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzGZPou/build/release/bin/kudu as pid 32275
I20260619 10:30:07.299710 9577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzGZPou/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781864754199919-9577-0/minicluster-data/master-2/wal/instance
I20260619 10:30:07.299908 9577 external_mini_cluster.cc:1428] Setting key 85713755e4c2c21885b6ced7a6549737
I20260619 10:30:07.300529 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.292594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.9.90.126:36985)
0619 10:30:07.292661 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.292663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.292835 (+ 172us) server_negotiation.cc:407] Connection header received
0619 10:30:07.293332 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.293333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.293342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.293399 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.294044 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.294520 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.295480 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.295682 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.295734 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.297850 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.297852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.297852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.297862 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.299124 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.299500 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.299501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.299501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.299549 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.299834 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.299836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.299836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.299962 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.300042 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.300352 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.300434 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 10:30:07.300736 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.292553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.126:36985 (local address 127.0.0.1:48208)
0619 10:30:07.292716 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.292756 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:07.293247 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:07.293470 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:07.293473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:07.293713 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:07.293969 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.293974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.294598 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.294599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.295401 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.295403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.295760 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.295761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.295859 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.296326 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:07.296336 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:07.297731 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:07.299199 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.299206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.299208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.299433 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.299610 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.299610 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.299611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.299755 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.300034 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:07.300042 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:07.300191 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 10:30:07.300304 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 10:30:07.300469 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260619 10:30:07.301308 32343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
I20260619 10:30:07.307559 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.301526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.125:46253 (local address 127.0.0.1:57668)
0619 10:30:07.301570 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.301579 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:07.301615 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:07.301774 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:07.301774 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:07.301916 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:07.301965 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.301967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.302568 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.302569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.303055 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.303057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.303390 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.303391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.303450 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.303822 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:07.303827 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:07.304976 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:07.306591 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.306592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.306592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.306795 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.306967 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.306968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.306968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.307026 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.307269 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:07.307270 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:07.307312 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 10:30:07.307372 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 10:30:07.307460 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 10:30:07.307559 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.301597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.9.90.125:46253)
0619 10:30:07.301666 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.301669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.301679 (+ 10us) server_negotiation.cc:407] Connection header received
0619 10:30:07.301701 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.301706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.301715 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.301761 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.302035 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.302497 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.303130 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.303331 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.303383 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.305063 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.305064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.305065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.305072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.306516 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.306861 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.306861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.306862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.306907 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.307084 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.307085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.307085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.307199 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.307253 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.307407 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.307472 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 10:30:07.307883 32343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
I20260619 10:30:07.308476 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.300282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.9.90.124:42017)
0619 10:30:07.300575 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.300581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.300602 (+ 21us) server_negotiation.cc:407] Connection header received
0619 10:30:07.300624 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.300627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.300641 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.300726 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.301463 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.302277 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.302997 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.303312 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.303370 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.305588 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.305596 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.305596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.305603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.307188 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.307481 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.307485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.307485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.307536 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.307740 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.307741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.307741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.307903 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.308032 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.308132 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.308405 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":91,"threads_started":1}
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.124@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.126@KRBTEST.COM
I20260619 10:30:07.314366 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.308116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42430)
0619 10:30:07.308166 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.308175 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:07.308207 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:07.308652 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:07.308653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:07.308778 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:07.308802 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.308803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.309501 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.309502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.310039 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.310041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.310367 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.310368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.310413 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.310740 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:07.310744 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:07.311790 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:07.313391 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.313392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.313392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.313585 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.313780 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.313781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.313782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.313858 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.314082 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:07.314083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:07.314126 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 10:30:07.314190 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 10:30:07.314263 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 10:30:07.314438 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.308177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.9.90.124:42017)
0619 10:30:07.308559 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.308561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.308569 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:30:07.308588 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.308589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.308599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.308657 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.308871 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.309433 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.310107 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.310305 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.310357 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.311872 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.311874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.311874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.311881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.313292 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.313661 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.313662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.313663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.313717 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.313924 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.313925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.313925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.314019 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.314070 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.314227 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.314322 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":368,"thread_start_us":96,"threads_started":1}
I20260619 10:30:07.315869 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.309210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.9.90.126:36985)
0619 10:30:07.309244 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.309246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.309292 (+ 46us) server_negotiation.cc:407] Connection header received
0619 10:30:07.309359 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.309360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.309369 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.309414 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.309849 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.310295 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.310922 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.311108 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.311174 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.313407 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.313409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.313409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.313417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.314949 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.315229 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.315231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.315231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.315285 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.315415 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.315416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.315416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.315524 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.315682 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.315719 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.315785 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 10:30:07.316095 32343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07: Bootstrap starting.
I20260619 10:30:07.316984 32343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07: Neither blocks nor log segments found. Creating new log.
I20260619 10:30:07.317286 32343 log.cc:824] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07: Log is configured to *not* fsync() on all Append() calls
I20260619 10:30:07.318029 32343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07: No bootstrap required, opened a new log
I20260619 10:30:07.319607 32343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } }
I20260619 10:30:07.319730 32343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3165c9104a6f48128a1dbc3af5d7ef07, State: Initialized, Role: FOLLOWER
I20260619 10:30:07.319787 32343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } }
I20260619 10:30:07.320050 32350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.320111 32343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:30:07.320152 32350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.90.125@KRBTEST.COM
W20260619 10:30:07.321831 32361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:30:07.321880 32361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:30:07.324070 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.316496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.9.90.125:46253)
0619 10:30:07.316537 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.316539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.316547 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:30:07.316570 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.316571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.316582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.316649 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.317094 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.317693 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.318600 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.318883 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.318956 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.321009 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.321010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.321010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.321018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.322621 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.323008 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.323009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.323009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.323067 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.323281 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.323282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.323282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.323408 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.323557 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.323895 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.323966 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:30:07.322157 32275 master_runner.cc:186] Error getting master registration for 127.9.90.126:36985: OK, instance_id { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" instance_seqno: 1781865006993199 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:30:07.324868 32275 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:30:07.325489 9577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 10:30:07.326478 32167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a64ec77ba795429c947f09f94c8f654e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48224
I20260619 10:30:07.337219 32193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } attempt: 2
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865006, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.126@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
I20260619 10:30:07.344090 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.337594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42444)
0619 10:30:07.337657 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.337669 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:07.337718 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:07.337859 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:07.337877 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:07.338035 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:07.338079 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.338081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.338704 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.338705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.339204 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.339205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.339544 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.339545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.339602 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.340021 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:07.340031 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:07.341226 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:07.343000 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.343001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.343002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.343213 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.343407 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.343408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.343408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.343462 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.343788 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:07.343788 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:07.343834 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 10:30:07.343895 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 10:30:07.343983 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 10:30:07.344118 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.337675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.9.90.124:42017)
0619 10:30:07.337725 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.337727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.337754 (+ 27us) server_negotiation.cc:407] Connection header received
0619 10:30:07.337779 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.337780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.337810 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.337854 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.338163 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.338618 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.339288 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.339474 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.339528 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.341328 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.341330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.341331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.341338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.342913 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.343285 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.343286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.343287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.343345 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.343519 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.343520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.343520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.343724 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.343793 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.343937 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.344025 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 10:30:07.345817 32193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e: Bootstrap starting.
I20260619 10:30:07.346563 32193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e: Neither blocks nor log segments found. Creating new log.
I20260619 10:30:07.346863 32193 log.cc:824] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e: Log is configured to *not* fsync() on all Append() calls
I20260619 10:30:07.347739 32193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e: No bootstrap required, opened a new log
I20260619 10:30:07.349156 32193 raft_consensus.cc:348] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } }
I20260619 10:30:07.349299 32193 raft_consensus.cc:729] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64ec77ba795429c947f09f94c8f654e, State: Initialized, Role: FOLLOWER
I20260619 10:30:07.349385 32193 consensus_queue.cc:260] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } }
I20260619 10:30:07.349704 32193 sys_catalog.cc:564] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:30:07.349758 32167 raft_consensus.cc:482] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 10:30:07.349689 32364 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.349843 32167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:30:07.349860 32364 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:07.350518 32167 raft_consensus.cc:504] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } }
I20260619 10:30:07.350729 32167 leader_election.cc:288] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [CANDIDATE]: Term 1 election: Requested vote from peers 3e34e80981cb4a48ade0ca7966739a07 (127.9.90.125:46253), 3165c9104a6f48128a1dbc3af5d7ef07 (127.9.90.124:42017)
W20260619 10:30:07.351529 32375 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:30:07.351596 32375 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:30:07.351861 32317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a64ec77ba795429c947f09f94c8f654e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3165c9104a6f48128a1dbc3af5d7ef07"
I20260619 10:30:07.351979 32317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:30:07.352813 32317 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64ec77ba795429c947f09f94c8f654e in term 1.
I20260619 10:30:07.353026 32142 leader_election.cc:302] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3165c9104a6f48128a1dbc3af5d7ef07, a64ec77ba795429c947f09f94c8f654e; no voters:
I20260619 10:30:07.353145 32364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 10:30:07.353293 32364 raft_consensus.cc:686] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [term 1 LEADER]: Becoming Leader. State: Replica: a64ec77ba795429c947f09f94c8f654e, State: Running, Role: LEADER
I20260619 10:30:07.353379 32364 consensus_queue.cc:237] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } }
I20260619 10:30:07.354187 32381 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a64ec77ba795429c947f09f94c8f654e. Latest consensus state: current_term: 1 leader_uuid: "a64ec77ba795429c947f09f94c8f654e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.354311 32381 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:07.354281 32125 master_runner.cc:186] Error getting master registration for 127.9.90.125:46253: OK, instance_id { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" instance_seqno: 1781865007133831 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 10:30:07.355841 32125 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 10:30:07.364149 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.356528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.9.90.125:46253)
0619 10:30:07.356563 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.356565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.357115 (+ 550us) server_negotiation.cc:407] Connection header received
0619 10:30:07.357221 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.357222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.357232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.357260 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.357849 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.358574 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.359654 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.359944 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.359985 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.361413 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.361414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.361414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.361422 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.362931 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.363310 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.363312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.363312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.363370 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.363567 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.363568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.363568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.363681 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.363745 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.363986 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.364062 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 10:30:07.365911 32268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } attempt: 3
I20260619 10:30:07.366122 32383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 10:30:07.366374 32383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 10:30:07.370816 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.356144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.9.90.126:36985)
0619 10:30:07.356181 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.356183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.356530 (+ 347us) server_negotiation.cc:407] Connection header received
0619 10:30:07.356693 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.356694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.356716 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.356746 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.360144 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.361150 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.365461 (+ 4311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.365772 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.365856 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.367957 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.367958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.367958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.367965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.369404 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.369759 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.369760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.369760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.369834 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.370258 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.370259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.370260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.370386 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.370465 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.370663 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.370732 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 19 10:30:07 dist-test-slave-hw7q krb5kdc[32112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781865007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.90.125@KRBTEST.COM for kudu/127.9.90.124@KRBTEST.COM
I20260619 10:30:07.373009 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.356403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.9.90.124:42017)
0619 10:30:07.356451 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.356453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.356759 (+ 306us) server_negotiation.cc:407] Connection header received
0619 10:30:07.356846 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.356847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.356856 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.356882 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.358146 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.358807 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.363424 (+ 4617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.363708 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.363743 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.365975 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.365977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.365977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.365983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.369247 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.371954 (+ 2707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.371955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.371956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.372016 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.372300 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.372301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.372301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.372445 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.372521 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.372833 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.372905 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 10:30:07.374639 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.366138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.90.124:42017 (local address 127.0.0.1:42466)
0619 10:30:07.366396 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 10:30:07.366406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 10:30:07.366462 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 10:30:07.366648 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 10:30:07.366649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 10:30:07.366798 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 10:30:07.366837 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.366839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.367769 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.367770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.368507 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 10:30:07.368510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.369782 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.369806 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 10:30:07.369859 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.370204 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 10:30:07.370209 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 10:30:07.371502 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 10:30:07.373623 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.373624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.373624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.373838 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.374028 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 10:30:07.374029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 10:30:07.374030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 10:30:07.374090 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 10:30:07.374345 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 10:30:07.374345 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 10:30:07.374388 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 10:30:07.374461 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 10:30:07.374555 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 10:30:07.374660 32317 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [term 1 FOLLOWER]: Refusing update from remote peer a64ec77ba795429c947f09f94c8f654e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:30:07.375005 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:07.366179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.9.90.124:42017)
0619 10:30:07.366228 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.366230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.366450 (+ 220us) server_negotiation.cc:407] Connection header received
0619 10:30:07.366510 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.366511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.366532 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.366603 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.366917 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.367661 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.369347 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:07.369662 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:07.369720 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:07.371593 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 10:30:07.371595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 10:30:07.371595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 10:30:07.371602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 10:30:07.373541 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.373906 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.373907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.373907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.373965 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 10:30:07.374149 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 10:30:07.374150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 10:30:07.374150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 10:30:07.374268 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 10:30:07.374319 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:07.374815 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 10:30:07.374913 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 10:30:07.375211 32381 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:30:07.376482 32268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07: Bootstrap starting.
I20260619 10:30:07.377064 32350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a64ec77ba795429c947f09f94c8f654e. Latest consensus state: current_term: 1 leader_uuid: "a64ec77ba795429c947f09f94c8f654e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.377146 32350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:07.377238 32268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07: Neither blocks nor log segments found. Creating new log.
I20260619 10:30:07.377568 32268 log.cc:824] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07: Log is configured to *not* fsync() on all Append() calls
I20260619 10:30:07.383159 32381 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a64ec77ba795429c947f09f94c8f654e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.383263 32381 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:07.383791 32383 catalog_manager.cc:1347] Generated new cluster ID: 52a4df3fe3994a77a4e41c64f5219ad1
I20260619 10:30:07.383836 32383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 10:30:07.383805 32350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a64ec77ba795429c947f09f94c8f654e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.383872 32350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:07.384181 32388 mvcc.cc:205] Tried to move back new op lower bound from 7298519070204182528 to 7298519070120091648. Current Snapshot: MvccSnapshot[applied={T|T < 7298519070204182528}]
I20260619 10:30:07.390543 32268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07: No bootstrap required, opened a new log
I20260619 10:30:07.391081 32383 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 10:30:07.392179 32383 catalog_manager.cc:1504] Loading token signing keys...
I20260619 10:30:07.392688 32268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } }
I20260619 10:30:07.392812 32268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e34e80981cb4a48ade0ca7966739a07, State: Initialized, Role: FOLLOWER
I20260619 10:30:07.392884 32268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } }
I20260619 10:30:07.393190 32268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 10:30:07.393299 32241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 10:30:07.395550 32241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [term 1 FOLLOWER]: Refusing update from remote peer a64ec77ba795429c947f09f94c8f654e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 10:30:07.395749 32242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a64ec77ba795429c947f09f94c8f654e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3e34e80981cb4a48ade0ca7966739a07"
I20260619 10:30:07.396423 32242 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64ec77ba795429c947f09f94c8f654e in term 1.
I20260619 10:30:07.396727 32381 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 10:30:07.399039 32364 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a64ec77ba795429c947f09f94c8f654e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.399122 32364 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e [sys.catalog]: This master's current role is: LEADER
I20260619 10:30:07.399024 32383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a64ec77ba795429c947f09f94c8f654e: Generated new TSK 0
I20260619 10:30:07.399284 32383 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 10:30:07.399969 32408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 10:30:07.400066 32408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 10:30:07.400202 32407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a64ec77ba795429c947f09f94c8f654e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.400308 32407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:07.400383 32407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a64ec77ba795429c947f09f94c8f654e. Latest consensus state: current_term: 1 leader_uuid: "a64ec77ba795429c947f09f94c8f654e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.400471 32407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:07.400537 32407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a64ec77ba795429c947f09f94c8f654e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64ec77ba795429c947f09f94c8f654e" member_type: VOTER last_known_addr { host: "127.9.90.126" port: 36985 } } peers { permanent_uuid: "3e34e80981cb4a48ade0ca7966739a07" member_type: VOTER last_known_addr { host: "127.9.90.125" port: 46253 } } peers { permanent_uuid: "3165c9104a6f48128a1dbc3af5d7ef07" member_type: VOTER last_known_addr { host: "127.9.90.124" port: 42017 } } }
I20260619 10:30:07.400614 32407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 10:30:07.401283 32405 mvcc.cc:205] Tried to move back new op lower bound from 7298519070267731968 to 7298519070120091648. Current Snapshot: MvccSnapshot[applied={T|T < 7298519070204182528}]
W20260619 10:30:07.423363 32196 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.421675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.9.90.126:36985)
0619 10:30:07.421730 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.421732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.422693 (+ 961us) server_negotiation.cc:407] Connection header received
0619 10:30:07.422794 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.422795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.422805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.422834 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.423286 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48248: BlockingRecv error: recv got EOF from 127.0.0.1:48248 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:30:07.424295 32348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.422071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.9.90.124:42017)
0619 10:30:07.422110 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.422112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.423793 (+ 1681us) server_negotiation.cc:407] Connection header received
0619 10:30:07.423885 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.423887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.423898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.423926 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.424233 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42480: BlockingRecv error: recv got EOF from 127.0.0.1:42480 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 10:30:07.425105 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.422399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.9.90.125:46253)
0619 10:30:07.422438 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.422439 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.424622 (+ 2183us) server_negotiation.cc:407] Connection header received
0619 10:30:07.424713 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.424714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.424725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.424752 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.425046 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57706: BlockingRecv error: recv got EOF from 127.0.0.1:57706 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 10:30:07.427449 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.426743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.9.90.125:46253)
0619 10:30:07.426787 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.426789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.426960 (+ 171us) server_negotiation.cc:407] Connection header received
0619 10:30:07.427061 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.427062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.427079 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.427105 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.427407 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57714: BlockingRecv error: recv got EOF from 127.0.0.1:57714 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 10:30:07.430442 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.429707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.9.90.125:46253)
0619 10:30:07.429738 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.429739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.429993 (+ 254us) server_negotiation.cc:407] Connection header received
0619 10:30:07.430080 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.430081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.430090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.430116 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.430402 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57724: BlockingRecv error: recv got EOF from 127.0.0.1:57724 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 10:30:07.434333 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.433612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.9.90.125:46253)
0619 10:30:07.433644 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.433646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.433854 (+ 208us) server_negotiation.cc:407] Connection header received
0619 10:30:07.433951 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.433951 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.433962 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.433989 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.434296 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57740: BlockingRecv error: recv got EOF from 127.0.0.1:57740 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 10:30:07.440263 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.439527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.9.90.125:46253)
0619 10:30:07.439564 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.439566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.439769 (+ 203us) server_negotiation.cc:407] Connection header received
0619 10:30:07.439857 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.439858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.439868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.439894 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.440216 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57748: BlockingRecv error: recv got EOF from 127.0.0.1:57748 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 10:30:07.450949 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.449920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.9.90.125:46253)
0619 10:30:07.449965 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.449967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.450284 (+ 317us) server_negotiation.cc:407] Connection header received
0619 10:30:07.450387 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.450388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.450400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.450429 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.450876 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57756: BlockingRecv error: recv got EOF from 127.0.0.1:57756 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:30:07.469520 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.468578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.9.90.125:46253)
0619 10:30:07.468624 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.468627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.468899 (+ 272us) server_negotiation.cc:407] Connection header received
0619 10:30:07.469007 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.469008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.469020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.469049 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.469459 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57762: BlockingRecv error: recv got EOF from 127.0.0.1:57762 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:30:07.504036 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.503133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.9.90.125:46253)
0619 10:30:07.503182 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.503184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.503443 (+ 259us) server_negotiation.cc:407] Connection header received
0619 10:30:07.503541 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.503542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.503555 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.503584 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.503975 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57770: BlockingRecv error: recv got EOF from 127.0.0.1:57770 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 10:30:07.570407 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.569626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.9.90.125:46253)
0619 10:30:07.569727 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.569729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.569740 (+ 11us) server_negotiation.cc:407] Connection header received
0619 10:30:07.569881 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.569882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.569896 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.569959 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.570339 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57778: BlockingRecv error: recv got EOF from 127.0.0.1:57778 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 10:30:07.700203 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.699451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.9.90.125:46253)
0619 10:30:07.699557 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.699559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.699642 (+ 83us) server_negotiation.cc:407] Connection header received
0619 10:30:07.699702 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.699703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.699715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.699766 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.700142 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57782: BlockingRecv error: recv got EOF from 127.0.0.1:57782 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 10:30:07.957935 32272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 10:30:07.957296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.9.90.125:46253)
0619 10:30:07.957391 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:07.957392 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:07.957413 (+ 21us) server_negotiation.cc:407] Connection header received
0619 10:30:07.957446 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:07.957447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:07.957457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:07.957501 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 10:30:07.957874 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57798: BlockingRecv error: recv got EOF from 127.0.0.1:57798 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 10:30:08.322288 32361 catalog_manager.cc:1259] Loaded cluster ID: 52a4df3fe3994a77a4e41c64f5219ad1
I20260619 10:30:08.322358 32361 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07: loading cluster ID for follower catalog manager: success
I20260619 10:30:08.323184 32361 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07: acquiring CA information for follower catalog manager: success
I20260619 10:30:08.323468 32361 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3165c9104a6f48128a1dbc3af5d7ef07: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:30:08.400543 32408 catalog_manager.cc:1259] Loaded cluster ID: 52a4df3fe3994a77a4e41c64f5219ad1
I20260619 10:30:08.400620 32408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07: loading cluster ID for follower catalog manager: success
I20260619 10:30:08.401453 32408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07: acquiring CA information for follower catalog manager: success
I20260619 10:30:08.401746 32408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e34e80981cb4a48ade0ca7966739a07: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 10:30:08.473552 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:08.470960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.9.90.125:46253)
0619 10:30:08.471061 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:08.471064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:08.471137 (+ 73us) server_negotiation.cc:407] Connection header received
0619 10:30:08.471204 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:08.471205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:08.471261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:08.471327 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:08.471479 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.471982 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.472776 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.472964 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.473020 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:08.473197 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:08.473317 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:08.473388 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:08.473442 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 10:30:08.473463 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":42,"threads_started":1}
I20260619 10:30:08.477614 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:08.475310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.9.90.124:42017)
0619 10:30:08.475429 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:08.475431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:08.475438 (+ 7us) server_negotiation.cc:407] Connection header received
0619 10:30:08.475478 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:08.475479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:08.475511 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:08.475578 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:08.475703 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.476183 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.476888 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.477086 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.477155 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:08.477294 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:08.477396 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:08.477460 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:08.477506 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 10:30:08.477526 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":47,"threads_started":1}
I20260619 10:30:08.481666 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:08.479145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.9.90.126:36985)
0619 10:30:08.479252 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:08.479254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:08.479262 (+ 8us) server_negotiation.cc:407] Connection header received
0619 10:30:08.479301 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:08.479302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:08.479349 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:08.479429 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:08.479516 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.480030 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.480851 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.481066 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.481128 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:08.481362 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:08.481462 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:08.481519 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:08.481560 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 10:30:08.481584 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260619 10:30:08.481750 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:08.479095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.9.90.125:46253)
0619 10:30:08.479244 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:08.479245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:08.479317 (+ 72us) server_negotiation.cc:407] Connection header received
0619 10:30:08.479360 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:08.479360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:08.479371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:08.479428 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:08.479557 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.480015 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.481083 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.481296 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.481356 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:08.481478 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:08.481507 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:08.481550 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:08.481650 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 10:30:08.481671 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260619 10:30:08.482620 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 10:30:08.479242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.9.90.124:42017)
0619 10:30:08.479274 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 10:30:08.479275 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 10:30:08.480084 (+ 809us) server_negotiation.cc:407] Connection header received
0619 10:30:08.480107 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 10:30:08.480108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 10:30:08.480116 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 10:30:08.480164 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 10:30:08.480265 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.480971 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.481952 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 10:30:08.482214 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 10:30:08.482279 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 10:30:08.482405 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 10:30:08.482435 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 10:30:08.482477 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 10:30:08.482536 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 10:30:08.482555 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 10:30:08.488159 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 32125
I20260619 10:30:08.492627 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 32200
I20260619 10:30:08.496657 9577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzGZPou/build/release/bin/kudu with pid 32275
2026-06-19T10:30:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2686 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5373 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (254303 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST